Mini Case Study: An Iterative Approach to Decoupling Drupal

If you're reading this, you may have already noticed that we've recently given Bounteous.com a fresh coat of paint. What might be less obvious is that we also took this redesign as an opportunity to slowly begin decoupling the front end of our existing Drupal site. We've considered decoupling in the past but were unable to justify the effort for a full-scale overhaul of our front end given other competing responsibilities. So what changed this time?

Our initial design concepts implied a phased approach. There were effectively only two pages that featured a completely new design and also incorporated a number of new behaviors and animations not present on the existing site. For this first phase, the rest of the site would get a mostly cosmetic overhaul, applying updated global styles to better match the new design introduced elsewhere on the site.

This put us at a similar crossroads. We believed that leveraging a JavaScript framework would greatly benefit our ability to achieve the motion-based interactions implied by our ambitious new designs. But for this phase, introducing a JavaScript framework wasn't really necessary for the rest of the site. In the short term, the cost of decoupling the entire site would greatly delay our ability to ship what was essentially just two new pages. This conclusion led us to a question that in hindsight seems pretty obvious:

Could we start by only decoupling two pages on our existing site?

Initially, we didn't know. But as we started considering this project with the assumption that we could make this change only for these two new pages, it became the difference between taking this step now or continuing to kick a large-scale change to our front-end architecture down the road to some undetermined date.

We eventually landed on an iterative approach to decoupling Bounteous' existing Drupal site with Gatsby; starting with only two pages, but laying the groundwork for any page on the site to be rendered primarily by either React or Twig. What follows is a look at how we did it, what we learned, and what we think this means for the future of our site.

One Site, Multiple Front Ends

For our JavaScript framework, we selected React, which we were technically already using in some minor ways on the existing site. While it would be possible to do this with a different framework, we found that the large React ecosystem would greatly accelerate our ability to achieve some of the motion-based interactions implied by our new designs. We ended up using both Framer Motion and react-lottie extensively, and they saved us quite a bit of time and effort.

While we had already decided that we'd be building additional React components in support of this new design concept, we also decided that we'd specifically be using Gatsby as our React framework of choice. Gatsby's plugin ecosystem greatly simplified the process of sourcing data from our existing Drupal CMS. Gatsby also opened up the possibility of statically generating portions of our site, which Bounteous.com was well suited for, given that most of our content changes infrequently.

Compared to a client-side approach to decoupling, server-side pre-rendering can have both SEO and performance benefits. As an added bonus, having these pages pre-rendered separately from Drupal also made it easier for React developers to contribute without ever having to set up a local Drupal environment.

Settling on these initial conclusions provided us with the following high-level architecture:

Drupal would be the CMS backend powering content for all of the pages on the site; both traditional CMS rendered pages, and pages rendered statically by the Gatsby build process. In the middle would be what we referred to as our 'front end globals.' These globals would be consumed by each front end and included shared styles, variables that serve as design tokens, along with full React components.

This structure allows us to take a progressive approach to introduce static content to our site. Initially, we'd only be building a small number of pages statically, but as we prove that this workflow can suit our site and our team, we could gradually shift where the line exists between the pre-built and dynamically built portions of our site.

Or alternatively, if we found that this approach didn't meet our needs, we could shift back to having Drupal render the content given that all of the data already exists in the CMS.

Front End Structure

After some consideration, we decided to take a monorepo style approach and have Gatsby, Drupal, and our front-end globals live in a single repository. Since this was a single domain and we had no concrete plans to distribute these components beyond Bounteous.com, we decided that a simplified repository would help streamline the process as we worked toward a tight timeline.

From the front-end perspective, this resulted in three main top-level directories in the repository: /fe-global, /drupal, and /gatsby. For this phase of the project, /fe-global exclusively contained Sass partials containing design tokens and global styles. Drupal and Gatsby would each selectively import from these partials as needed.

On the React side, we initially focused on building functional components with as little internal state as possible. This allowed us to prototype in the browser early, and also would allow us to provide data to these components from various contexts.

Regardless of if the data was being sourced from Gatsby's GraphQL API, directly from Drupal, or even hardcoded, the same component could be used. This also allowed us to use Storybook heavily during this phase of the project in order to get early feedback on these components before data was fully integrated.

On the Drupal side of things, we created new content types for each of our decoupled page templates. We also continued to use paragraphs to represent our components as we had been doing for existing content on the site.

The structure of data from the Paragraphs module initially doesn't seem like a natural fit for decoupled Drupal projects, but with gatsby-source-drupal and a few small utilities (which we'll talk about later), we found this data to be reasonable to deal with. In fact, it ended up giving us a high level of layout control, down to the ability to reorder components on the resulting static pages.

Considering that the majority of our content was still being rendered by Drupal, we still had our traditional Drupal theme. This theme incorporated the partials and tokens from our front-end globals alongside Drupal-specific styles, templates, and JavaScript.

Serving a Subset of Decoupled Pages

One of the very first things we had to prove out to ensure that this approach was feasible was serving a combination of static routes (pre-rendered by Gatsby) alongside dynamic routes handled by Drupal.

As part of our Gatsby build process, we are copying the 'public' directory which represents Gatsby's build asset, into the document root for our Drupal site. For the initial phase of this project, we were able to use a couple of very specific .htaccess rules to serve our two new static routes.

We knew this solution wouldn't scale long term as we introduced more content to our site. Ideally, we'd want to be able to create Decoupled content within Drupal, specify a path alias, and automatically have that route handled statically. We eventually found that we could achieve this via .htaccess as well.

Our rules take advantage of Drupal's URLs not having a "file" component to them. When we call createPages in gatsby-node.js with an alias like /services, gatsby creates that route as /services/index.html. The main .htaccess rule checks if /public/<url>/index.html exists and rewrites if it does.

This essentially means that for any request the Gatsby route 'wins' if there is a related file in the 'public' directory (/public/my-alias/index.html, for example,) and all other requests fall back to being handled by Drupal. This has the extra advantage of bypassing Drupal's bootstrap process for all of our static routes.

As focus shifted over to data integration, some adjustments were also necessary to configure the gatsby-source-drupal plugin to meet our needs. The gatsby-source-drupal plugin pulls data from Drupal's JSON:API endpoints and makes this data available to React components via Gatsby's GraphQL API. By default, the plugin imports all data from the source Drupal site. Since for this initial phase Gatsby would only be used to build a small subset of pages, most of this data was unnecessary and also would have the side effect of greatly increasing our build times.

As an initial attempt to solve this problem, we used Drupal's JSON:API Extras module to only expose the resources that our Gatsby build needed to depend on. This helped, but we still eventually needed to enable the file resource, which pretty much immediately sunk our build times.

Gatsby was now importing (and worse yet processing) local versions of years worth of images that we didn't need to support our new content. We eventually found that it was possible to configure gatsby-source-drupal to only import the files referenced by content that was necessary for our builds, but it required a combination of configuration options that wasn't completely obvious from the documentation.

The first step was to add the file resource as a disallowed link type:

// In your gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: 'gatsby-source-drupal',
      options: {
        baseUrl: <your_url>,
        // Disallow the full files endpoint
        disallowedLinkTypes: ['self', 'describedby', 'file--file'],
      },
    },
  ],
}

This alone would result in all files being ignored by the plugin. A little bit further on in the disallowed link types documentation is the following note:

When using includes in your JSON:API calls the included data will automatically become available to query, even if the link types are skipped using disallowedLinkTypes. This enables you to fetch only the data you need at build time, instead of all data of a certain entity type or bundle.

This essentially allows us to re-include specific files if they are referenced by other content. What makes this feature potentially easy to miss is the fact that it uses the plugin's filter option, which typically further restricts the data sourced from the plugin. The resulting configuration ended up looking like this:

// In your gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: 'gatsby-source-drupal',
      options: {
        baseUrl: <your_url>,
        /// Disallow the full files endpoint
        disallowedLinkTypes: ['self', 'describedby', 'file--file'],
        filters: {
          // Use includes so only the files associated with our decoupled content
          // types are included.
          "paragraph--dhp_hero": "include=field_dhp_fg_img",
          "paragraph--dhp_animation_cards": "include=field_dhpac_images",
          "paragraph--featured_post": "include=field_dfp_bg_img",
        },
      },
    },
  ],
}

With this configuration, if a featured post paragraph is used on the homepage, any associated background images (field_dfp_bg_img) will be sourced by Gatsby as well.

Providing Drupal Data to Our React Components

So at this point, we have access to all of the necessary data, and also a set of functional components that aren't yet aware of Drupal's data. We also have content types that can use a number of different paragraph types, in any order. This is great from the perspective of layout flexibility, but less predictable from a data integration standpoint.

To help manage this mapping we created a custom React utility called paragraphsToComponents. Assuming that we have an existing GraphQL query that provides paragraph data to our template component, we could use it like this:

const HomePage = ({ data }) => {
  const paragraphs =
    data.nodeDecoupledHomePage.relationships.field_dhp_components

  const paragraphComponents = useParagraphsToComponents(paragraphs)

  return (
      <DefaultLayout expanded={true}>
      {paragraphComponents.map((paragraph, index) => {
          return (
            <div key={'paragraph_${index}'}>
              {paragraph.provider({
                paragraph: paragraph,
                index: index,
              })}
            </div>
          )
        })}
      </DefaultLayout>
  )
}

As we'll see in a second, the utility returns an array of components that can be used to render the related paragraph data. In the template component's render method we iterate through this array and render these paragraphs in order. This allows us to correctly process paragraph data in any order, with little heavy lifting or redundant code in our template components.

The utility itself is defined as follows:

import AboutUsBannerProvider from "../components/paragraphs/provider/AboutUsBannerProvider"
import AnimationProvider from "../components/paragraphs/provider/AnimationProvider"
import CalloutProvider from "../components/paragraphs/provider/CalloutProvider"
// … Additional component imports ...

// The paragraphs above map to these components:
const componentMap = {
  dhp_about_us_banner: AboutUsBannerProvider,
  dhp_animation_cards: AnimationProvider,
  dhp_callout: CalloutProvider,
  // … Additional component mappings ...
}

const paragraphsToComponents = paragraphs => {
  // Create a new array with paragraph data that also specifies the React component
  // we'll use to render it.
  const mappedParagraphs = paragraphs
    // Add a component key that defines the component using the following
    // naming convention: ParagraphTypeProvider
    .map(paragraph => {
      const componentType = paragraph.__typename.replace("paragraph__", "")
      paragraph.provider = componentMap[componentType]
      return paragraph
    })
    // Filter out paragraph types we don't yet have a mapped component for.
    .filter(paragraph => {
      return paragraph.provider !== undefined
    })
  return mappedParagraphs
}

export default paragraphsToComponents

This assumes a particular naming convention for our components: ParagraphTypeProvider where 'ParagragraphType' matches the Paragraph Type name from Drupal.

As you can see in the example below, our Provider components only have one responsibility: providing the appropriate data from Drupal to our functional components.

import React from "react"

import Callout from "../../components/Callout/Callout"
import HeadlineDivider from "../../components/HeadlineDivider/HeadlineDivider"
import { graphql } from "gatsby"

const CalloutProvider = ({ paragraph }) => {
  const heading = paragraph?.field_dhpc_heading?.processed
  const body = paragraph?.field_dhpc_copy?.processed
  const backgroundOption = paragraph.field_dhpc_bg_opts

  if (backgroundOption === "background__waves") {
    return <HeadlineDivider heading={heading} body={body} />
  } else {
    return <Callout heading={heading} body={body} />
  }
}

export default CalloutProvider

export const CalloutFragment = graphql'
  fragment CalloutFragment on paragraph__dhp_callout {
    id
    field_dhpc_bg_opts
    field_dhpc_callout_size
    field_dhpc_copy {
      processed
    }
    field_dhpc_heading {
      processed
    }
  }
'

We're also defining a GraphQL fragment for the data that is required for this component. This gives us a consistent definition of the necessary Callout Paragraph data that can be imported into any other component that needs it.

Getting to this point took a decent amount of time and effort, but once defined it became much easier to integrate future Paragraphs from Drupal by following this pattern.

Shared React Components

There also was an integration problem to solve going from React to Drupal. We needed to syndicate the same header and footer component used by Gatsby to our Drupal pages so that we could provide a consistent look and feel throughout the site, regardless of which front-end technology owned rendering that page. Thankfully, by ensuring that our React components were strictly presentational we were well suited to use these components in a different context.

We approached this by creating an "exports" subdirectory alongside the rest of our React components which contained an exportable version of the header and footer. These essentially functioned as provider components just as we saw with our Gatsby data integration. Initially, these exported components used pre-defined data since they didn't have access to Gatsby's GraphQL API. However, we eventually found a solution to export these components using the same data that is available to our Gatsby build.

As a first step, we created a separate Webpack configuration that used these two components as entry points, and placed the related bundles into a 'dist' directory in the Drupal theme.

On the Drupal side, we used the Component module to help ease this integration. As a simplified successor to the progressively decoupled blocks module, Component allows you to define configuration in a .yml file alongside your JavaScript in order to expose your component to Drupal. In the case of our navigation, we defined the following configuration:

name: Evolution Navbar
description: 'Evolution Navbar'
type: 'block'
js:
  'dist/navbar.bundle.js' : {}
  'dist/vendors~navbar.bundle.js': {}
dependencies:
  - bounteous/react
template: 'evolutionnavbar.html'
form_configuration:
  theme:
    type: select
    title: "Theme"
    options:
      '': 'Dark Theme'
      'theme-light': 'Light Theme'
    default_value: ''

Alongside the following template:

<div id="evolution-navbar"></div>

This manages to do quite a lot with a little. Based on this configuration, a new block will be created that uses our evolutionnavbar.html template and loads our component JavaScript and any dependencies as a library. It also exposes a configuration form which in this case allows you to specify a light or dark theme to be used when rendering the component. The values of any form configuration will be added to the template as data attributes, in this case making 'data-theme' available to our component.

With that in place, the code for the navbar React component that we'll be exporting is as follows:

import React from "react"
import ReactDOM from "react-dom"

import { MediaContextProvider } from "../../components/layouts/Media/Media"
import abstracts from "styles/abstracts.scss"

// Grab cached data from disk.
import NavigationProvider from "../../components/provider/menu/NavigationProvider"
import templateData from "../../../public/page-data/template/page-data.json"

const queryHash = templateData.staticQueryHashes[1]
const data = require('../../../public/page-data/sq/d/${queryHash}.json')

const drupalProvider = document.querySelector(".evolutionnavbar")
const config = drupalProvider.dataset

ReactDOM.render(
  <MediaContextProvider>
    <NavigationProvider
      data={data.data}
      color={abstracts.cPlum}
      theme={config.theme}
    />
  </MediaContextProvider>,
  document.getElementById("evolution-navbar")
)

First, we're importing cached menu data from our Gatsby build. This was inspired by the approach outlined in Exporting an embeddable React component from a Gatsby app using Webpack. Sourcing this from a cache using a query hash seems…fragile, but it has been reliable for our needs thus far. This assumes that the Gatsby build runs prior to the Drupal build, which already happened to be the case in this project.

Next, we're selecting the wrapping div in the DOM in order to access all of the data attributes provided by our Drupal block. This allows us to pass the theme option set in the instance of this block as a prop to our navigation component.

Finally, we mount the component into #evolution-navbar which is used in the template that was specified in our block configuration.

This approach could add some overhead as the number of components you're working with increases, but works nicely for our header and footer. It also allows us to easily configure different instances of the component block to be used on different sections of the site. We use this to swap between the dark and light themes of the header, and even specify if the form in our footer should be expanded or collapsed by default.

Looking Ahead

While we're happy with the progress made with this initial release, there is intentionally more evolution to come. We've been working on improving the content editing and deployment process introduced by this new workflow. These changes include configuring Gatsby live preview, and also making enhancements to the build hooks module to allow our content team to trigger Gatsby builds on demand.

As we've continued incorporating more content into our Gatsby build process, we've also run into some pain and confusion around the seemingly arbitrary divide between our React and Twig components. In addition to working to make this distinction clearer to our team, we've also been experimenting with solutions that allow our React and Twig components to be used side by side in more contexts, including syndicating markup and styles from Drupal to be used in Gatsby content as needed.

So far we think this iterative approach can be of benefit to others looking to transition the front end of their existing Drupal platform without requiring the commitment of a large-scale re-architecture.

Taking an iterative approach can instead make it possible to prove that decoupling has clear value and also ensure that changes to the development, content editing, and deployment process fit the needs of your team. We're excited to continue evolving Bounteous.com and hope that you'll follow along.

For even more on this topic, check out Episode #284 of the Talking Drupal Podcast - Iterative Approach to Decoupling and An Iterative Approach To Decoupling Your Existing Drupal Site With Gatsby at DrupalCon North America.