How to share resources across CircleCI workflows

June 30, 2018

Workflows are a powerful method of managing multiple jobs across CircleCI’s 2.0 builds, enabling concurrent processes, sharing data across jobs, different machine types, etc. The benefits available to workflows are pretty awesome, but, for this post, lets look at how to share data across jobs to make our builds faster and more efficient. A perfect example for us is running bundle install once and making the gem files available to multiple concurrent runs of RSpec, Cucumber, or different Selenium runs without requiring each job to run that on its own.

Sounds useful right? Let’s take a look at how to achieve this with an example!

Example

Here’s our base config.yml with two jobs: rspec and cucumber. Both of them must run bundle install and if there’s no existing cache, they’ll both have to download everything needed. While this isn’t always an issue(like in this example), if you have a complex project with lot of set up needed or use larger dependencies like the AWS SDK, having to do this downloading/set up can take some time. For the example our tests are extremely basic. The code is here if you’d like to check it out.

So here’s the first iteration of the circle config.yml before being setup to share resources:

version: 2

aliases:
    - &restore_gem_cache
        name: Saving gem cache
        key: v1-gemfile-{{ checksum "Gemfile.lock" }}

    - &save_gem_cache
        name: Saving gem cache
        key: v1-gemfile-{{ checksum "Gemfile.lock" }}
        paths:
            - ~/data/vendor/bundle

    - &bundle_install
        name: Install Gems
        command: bundle install --path=vendor/bundle --jobs=4 --retry=3

defaults: &defaults
  docker:
    - image: ruby:2.5.1
  working_directory: ~/data

jobs:
  rspec:
    <<: *defaults
    steps:
      - checkout
      - restore_cache: *restore_gem_cache
      - run: *bundle_install
      - save_cache: *save_gem_cache
      - run: bundle exec rspec

  cucumber:
    <<: *defaults
    steps:
      - checkout
      - restore_cache: *restore_gem_cache
      - run: *bundle_install
      - save_cache: *save_gem_cache
      - run: bundle exec cucumber

workflows:
    version: 2
    pr:
        jobs:
            - rspec
            - cucumber

As we can see in the jobs, both have to restore_cache, run bundle install, and then save_cache before running what they need. If you’re curious, here’s what the above config looks like in Circle after it’s been run.

non-sharing

From the RSpec job, 01:04 was spent on downloading gem files and similarly, the Cucumber job took 0:54 to do the same. While they’re both doing this concurrently, it’s inefficient and if you have more involved setup, this could get unwieldy, quick. Why not separate out our setup and only do it once? Let’s see what that’d could look like with our example.

Before we take a look at the new config, here’s what the new behavior looks like in Circle.

non-sharing

For the curious, here’s how fast it ends up being when using cached gems:

non-sharing

And the config:

# omitted unchanged
...

jobs:
    bundle_dependencies:
        <<: *defaults
        steps:
            - checkout
            - attach_workspace:
                at: ~/data
            - restore_cache: *restore_gem_cache
            - run: *bundle_install
            - save_cache: *save_gem_cache
            - persist_to_workspace:
                root: .
                paths:
                    - vendor/bundle
    rspec:
        <<: *defaults
        steps:
            - checkout
            - attach_workspace:
                at: ~/data
            - run: bundle --path vendor/bundle
            - run: bundle exec rspec

    cucumber:
        <<: *defaults
        steps:
            - checkout
            - attach_workspace:
                at: ~/data
            - run: bundle --path vendor/bundle
            - run: bundle exec cucumber

workflows:
    version: 2
    pr:
        jobs:
            - bundle_dependencies
            - rspec:
                requires:
                    - bundle_dependencies
            - cucumber:
                requires:
                    - bundle_dependencies

Alright. There’s a few changes here, but nothing too crazy. Let’s go over each area that changed.

  1. New job: bundle_dependencies

    • This job is our gem installer now. It’ll handle restoring cache if present, downloading gems, etc. The important detail here though is the persist_to_workspace line. persist_to_workspace is a special step that tells Circle to take the files that we specify in the paths, put them in a location relative to our working directory, and save them to be used by another job in our workflow. Simply put, we can think of this step as uploading our files to a shared drive. For our example, we’re telling Circle to keep the vendor/bundle directory with all our installed gems available for our other jobs to use. In this simple use case, we’re just using our working directory, but you’re welcome to organize however you like or need to.
  2. Modify existing jobs rspec and cucumber

    • Since we have bundle_dependencies to handle our gems for us, we can remove the steps it handles. Yay for deleting code! We still have to run bundle --path vendor/bundle as these jobs aren’t the same machine as the one we installed the gems to but since we already have all the files it should be near instant. In order for that command to be successful however, we need to tell the job where to look for its files. We do that by telling the attach_workspace command where our files are. Look at how clean our jobs look now!
  3. Modify our workflow

    • Our jobs section is just a touch different in that we’ve added our bundle_dependencies job and have told Circle that our other jobs require it to finish first. By setting jobs as requirements to other jobs, you can get some really intricate gating going on in your builds to tailor things just how you want them.

So there you have it! Now you can share resources between jobs and craft pretty awesome Circle builds to fit your needs!

If I’ve helped you out or you have feedback or questions, please feel free to reach out. Thanks!

Js