Untitled RSS Feed

Faster acceptance tests

Adam Meadows — Wed, 24 Aug 2016 00:00:00 GMT

You may have noticed that when working with ember-cli-mocha, acceptance tests created with

ember g acceptance-test

Can run rather slowly if you use proper BDD ettiquette and include an it() for every expect(). This is due primarily to the fact that the application is created in a top-level beforeEach() and destroyed in a top-level afterEach() While this makes your tests very nicely independent, it can also make your tests rather slow, since the app is created and your route rendererd independently for each it().

If you’re like me, you often set up a scenario in your beforeEach() and then have a few different things you want to verify expected behavior for, without changing the state of your application. In cases like those, what you really want is before() and after() instead of beforeEach() and afterEach(). Inside a describe(), before() is executed once, then all the it() methods are executed, then the after() is executed once. On the other hand, as the name suggests beforeEach() is exectued before each individual it() method and afterEach() is executed after each individual it() method.

So, what happens when the following test is run?

import {afterEach, beforeEach, describe, it} from 'mocha'
import {expect} from 'chai'
import startApp from '../../../helpers/start-app'
import destroyApp from '../../../helpers/destroy-app'

describe('Acceptance: Login page shows form', function () {
  let application

  beforeEach(function () {
    application = startApp()
    visit('/login')
  })

  afterEach(function () {
    destroyApp(application)
  })

  it('should display the company logo', function () {
    expect(find('img.logo')).to.have.length(1)
  })

  it('should display a username input', function () {
    expect(find('input[type=text].username')).to.have.length(1)
  })

  it('should display a password input', function () {
    expect(find('input[type=password].password')).to.have.length(1)
  })
})

Well, since we’re using the default beforeEach() to perform the startApp() call, and we have 3 it() methods, the app is being created 3 times, and each time a single expectation is being verified.

Now, you may be thinking, if I want to speed that up, all I have to do is combine the it() methods. After all, beforeEach() will run once for each it(), so if I only want it to run once, I should just have a single it().

Seems logical… right?

Wrong.

Don’t do that.

No one wants to see this test:

import {afterEach, beforeEach, describe, it} from 'mocha'
import {expect} from 'chai'
import startApp from '../../../helpers/start-app'
import destroyApp from '../../../helpers/destroy-app'

describe('Acceptance: Login page shows form', function () {
  let application

  beforeEach(function () {
    application = startApp()
    visit('/login')
  })

  afterEach(function () {
    destroyApp(application)
  })

  it('should render the logo and login form', function () {
    expect(find('img.logo')).to.have.length(1)
    expect(find('input[type=text].username')).to.have.length(1)
    expect(find('input[type=password].password')).to.have.length(1)
  })
})

And here’s why:

It doesn’t read well.
Failures are difficult to track down

Sure, you can try to mitigate #2 by doing something like this:

import {afterEach, beforeEach, describe, it} from 'mocha'
import {expect} from 'chai'
import startApp from '../../../helpers/start-app'
import destroyApp from '../../../helpers/destroy-app'

describe('Acceptance: Login page shows form', function () {
  let application

  beforeEach(function () {
    application = startApp()
    visit('/login')
  })

  afterEach(function () {
    destroyApp(application)
  })

  it('should render the logo and login form', function () {
    expect(
      find('img.logo'),
      'the logo should be shown'
    ).to.have.length(1)

    expect(
      find('input[type=text].username'),
      'the username input should be shown'
    ).to.have.length(1)

    expect(
      find('input[type=password].password'),
      'the password input should be shown'
    ).to.have.length(1)
  })
})

But really, you can’t honestly tell me that’s easy to read/follow.

Instead, embrace the power of before() and speed up your test without losing readability or ease of debugging:

import {after, before, describe, it} from 'mocha'
import {expect} from 'chai'
import startApp from '../../../helpers/start-app'
import destroyApp from '../../../helpers/destroy-app'

describe('Acceptance: Login page shows form', function () {
  let application

  before(function () {
    application = startApp()
    visit('/login')
  })

  after(function () {
    destroyApp(application)
  })

  it('should display the company logo', function () {
    expect(find('img.logo')).to.have.length(1)
  })

  it('should display a username input', function () {
    expect(find('input[type=text].username')).to.have.length(1)
  })

  it('should display a password input', function () {
    expect(find('input[type=password].password')).to.have.length(1)
  })
})

Now what’s happening. Well, since before() runs before all it() methods, and after() runs after all it() methods, we have a single app instance being created, and then three expectations happening before the app instance is destroyed.

WARNING: Creating your app in before() means that all your nested functionality is using the same app, with the same state. So, if you click a check-box in one test, it’ll stay clicked for the next test unless you reset your state somehow!

The above limitation means you have two options when writing acceptance tests, since I don’t consider having to clean up after yourself after every interaction with your app a valid option.

The first option is to create a new describe() with it’s own startApp() and destroyApp() for each interaction you want to test. This is a valid option in many situations, but may not be as desirable in others.

The second option is to use another mechanism to reset your state to a known starting point before beginning to interact with your application. For instance, you could use visit() to navigate to another route (preferably a very easy to load one, like not-found) and then navigate back to the current route (effectively doing a refresh).

Using ember-cli-visual-acceptance

Eric White — Mon, 18 Jul 2016 00:00:00 GMT

Installation

Simple as running ember install ember-cli-visual-acceptance

Testem.json

In order for ember-cli-visual-acceptance to capture images, "launch_in_ci" must contain ethier SlimerJsVisualAcceptance or PhantomJsVisualAcceptance.

Example:

{
  "framework": "mocha",
  "test_page": "tests/index.html?hidepassed",
  "disable_watching": true,
  "launch_in_ci": [
    "SlimerJsVisualAcceptance"
  ],
  "launch_in_dev": [
    "Firefox"
  ],
  "launchers": {
    "PhantomJsVisualAcceptance": {
      "command": "phantomjs vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    },
    "SlimerJsVisualAcceptance": {
      "command": "slimerjs -jsconsole vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    }
  }
}

Usage

Inside an integration or acceptance test you can use return capture('') at the end of an it() function.

test.js

it('renders', function () {
  // Set any properties with this.set('myProperty', 'value')
  // Handle any actions with this.on('myAction', function (val) { ... })

  this.render(hbs`{{frost-file-picker}}`)
  expect(this.$('.frost-file-picker')).to.have.length(1)
  return capture('Basic-File-Picker')
})

You can also use this functionality for async testing:

it('renders', function (done) {
  // Set any properties with this.set('myProperty', 'value')
  // Handle any actions with this.on('myAction', function (val) { ... })

  this.render(hbs`{{frost-file-picker}}`)
  expect(this.$('.frost-file-picker')).to.have.length(1)
  capture('Basic-File-Picker').then(function (data) {
    console.log(arguments)
    done()
  }).catch(function (err) {
    done(err)
  })
})

In this case done comes from it('test', function (done) { … }. You can also use the then functionality to get creative rather than using done. arguments will display the response from ResembleJS or the message, 'No passed image. Saving current test as base' (If there is no current baseline image).

Usage with Qunit

To use with Qunit you must supply the assert parameter.

Example (Using ember-cli-showdown):

import hbs from 'htmlbars-inline-precompile';
import { moduleForComponent, test } from 'ember-qunit';

moduleForComponent('markdown-to-html', 'Integration | Component | markdown to html', {
  integration: true
});

test('positional parameter', function(assert) {
  assert.expect(1);
  this.set('markdown', '*hello world*');
  this.render(hbs`{{markdown-to-html markdown}}`);
  assert.equal(this.$('> *').html(), 'hello world');
  return capture('Markdown', {assert: assert});
});

Larger frame

In order to go beyond the default 640x340px dimensions you can supply the width and height to the capture function. The following shows an example to get a 1920x4041 container: capture('', {width: 1920, height: 4041})).

Targeting an element

ember-cli-visual-acceptance also has the ability to target a specific element. Given a DOM element ember-cli-visual-acceptance will capture the specified element rather than the default #ember-testing-container. To target an element you must pass in the DOM element.

Target with an id

capture('Markdown', { targetElement: document.getElementById('foo')});

Target with class selectors

capture('drop-down-container', { targetElement: this.$('.drop-down-container')[0]})

Setting up travis

Replace ember test with ember tva. This command comes with ember-cli-visual-acceptance and provides the functionality for commenting a report (Stored on Imgur) on a PR from the Travis build.

Add your github credentials to before_script:

before_script:
- git config --global user.email "ericwhite613@gmail.com"
- git config --global user.name "Eric White"
- git config --global push.default simple
- npm set progress=false
- git config credential.helper "store --file=.git/credentials"
- echo "https://${GH_TOKEN}:@github.com" > .git/credentials

Secure Variables (Display value in build log = OFF)

Add GH_TOKEN to Travis. Personal Access Token must have push access

Unsecure Variables (Display value in build log = ON)

Add RO_GH_TOKEN Unsecure token that can only read.
Add VISUAL_ACCEPTANCE_TOKEN token, value can be found here
If you put the VISUAL_ACCEPTANCE_TOKEN directly in your code and commit it to Github; Github will revoke the token. So it may be smart for you to create your own dummy Github account, and create a personal access token that has it’s repo scope set.

Pushing only when conditions are met

In some instances you may have multiple jobs with your Travis Builds. Wishing only to push in one. Ember-cli-visual-acceptance has added two options to ember tva, push-var and push-on. Using these two options visual acceptance will only push when push-var is equal to push-on. For instance we could do ember tva --push-var=$EMBER_TRY_SCENARIO --push-on=default, so visual acceptance will only push when doing the build job for EMBER_TRY_SCENARIO=default.

Playing nice with Pr-bumper or doing your own push

Using push-var and push-on we can also stop visual-acceptance from pushing and having only pr-bumper or something else push. This can be accomplished with the following: ember tva --push-var='false'

Github Enterprise

The api endpoint can be specified with the --github-api-endpoint="http(s)://hostname/api/v3" option

NightmareJS (Electron) vs Browsers - html2canvas vs. PhantomJS render callback

You must enable the display to use headless browsers by adding the following to the before_script hook:

before_script:
- "export DISPLAY=:99.0"
- "sh -e /etc/init.d/xvfb start"
- sleep 3 # give xvfb some time to start

NightmareJS (Electron)

Nightmare takes a screenshot with Electron and provides the most accurate image. Using nightmare requires the nightmare and nightmare-custom-event packages to be saved to your dev dependencies npm install nightmare nightmare-custom-event --save-dev (if using the visual acceptance blueprint this will be automatic)

If a captured image comes up blank or cutoff you should edit the vendor/nightmarejs-launcher.js file to increase the viewport (the default is .viewport(3000, 10000) width by height). The reason for this is Electron can only screenshot what it can see, and the test page continously grows with the more tests you have. This solution costs less than attempting to scroll the ember testing rectangle in frame, and resizing Electron to fit the image. Additionally, as long as Nightmare’s show option is not set to true you can get away with setting ludicrous sizes.

PhantomJS - SlimerJS

PhantomJS and SlimerJS can both be used with this tool to capture images.

Personally I prefer using SlimerJS as their version of Gecko matches the latest Firefox. While PhantomJS Webkit is about a year behind Safari’s Webkit version. SlimerJsVisualAcceptance images come out much more accurate. Additionally, debugging the images produced from the .ember-testing-container in Firefox is useful. Since the .ember-testing-container is identical in SlimerJS and Firefox ( at least I’ve never seen a difference between the two).

Warning

With certain repositories I’ve had trouble with SlimerJS having segmentation faults on both Linux and Mac. I’ve yet to resolve this issue.

Html2Canvas

Html2Canvas is used when a browser does not have the function window.callPhantom (Only PhantomJS and SlimerJS have this defined). Html2Canvas is still in beta and as result you will see some issues. Html2Canvas relies on Canvas drawing support. I find Chrome has the best Canvas drawing support (miles ahead of their competitors), while Firefox has the second best Canvas drawing support.

SVGs

Html2Canvas has difficulties rendering SVGs (more so in Firefox than in Chrome). As a result I have added a new expermental functionality that attempts to render the svgs better. You can use this experimental feature by setting experimentalSvgs to true (Example: capture('svg-experimental',{ experimentalSvgs: true})).

Experimental SVGs will not be used for PhantomJS, SlimerJS, and Chrome/Chromium (Chrome properly renders svgs with html2canvas) as their rendering handles SVGs (PhantomJS and SlimerJS basically just take a screenshot of the page)

Using Nightmare

Testem.json

{
  "framework": "mocha",
  "test_page": "tests/index.html?hidepassed",
  "disable_watching": true,
  "launch_in_ci": [
    "NightmareJsVisualAcceptance"
  ],
  "launch_in_dev": [
    "Firefox"
  ],
  "launchers": {
    "PhantomJsVisualAcceptance": {
      "command": "phantomjs vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    },
    "SlimerJsVisualAcceptance": {
      "command": "slimerjs -jsconsole vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    },
    "NightmareJsVisualAcceptance": {
      "command": "node vendor/nightmarejs-launcher.js ",
      "protocol": "browser"
    }
  }
}

Using Chromium

To use Chromium on Travis set

Testem.json

  "launch_in_ci": [
    "Chromium"
  ],

If you’re using Mac OS X you’ll have to remove this when testing locally.

Using Firefox

To use Firefox on Travis simply set

Testem.json

  "launch_in_ci": [
    "Firefox"
  ],

And add the following to your .travis.yml to get the latest version of Firefox:

addons:
  firefox: "latest"

Using SlimerJS

Testem.json

{
  "framework": "mocha",
  "test_page": "tests/index.html?hidepassed",
  "disable_watching": true,
  "launch_in_ci": [
    "SlimerJsVisualAcceptance"
  ],
  "launch_in_dev": [
    "Firefox"
  ],
  "launchers": {
    "PhantomJsVisualAcceptance": {
      "command": "phantomjs vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    },
    "SlimerJsVisualAcceptance": {
      "command": "slimerjs -jsconsole vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    }
  }
}

Using PhantomJS

Testem.json

{
  "framework": "mocha",
  "test_page": "tests/index.html?hidepassed",
  "disable_watching": true,
  "launch_in_ci": [
    "PhantomJsVisualAcceptance"
  ],
  "launch_in_dev": [
    "Safari"
  ],
  "launchers": {
    "PhantomJsVisualAcceptance": {
      "command": "phantomjs vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    },
    "SlimerJsVisualAcceptance": {
      "command": "slimerjs -jsconsole vendor/phantomjs-launcher.js ",
      "protocol": "browser"
    }
  }
}

Reports for multiple browsers

Producing a report for multiple browsers is perfectly fine. All you need to do is add your collection of browsers to launch_in_ci.

Example:

Testem.json

"launch_in_ci": [
    "Firefox",
    "SlimerJsVisualAcceptance"
  ],

`timeout of 2000ms exceeded`

With Travis you may find you see this error a few times. Sometimes Travis can take a while to run capture (especially if you’re using experimentalSvgs and have a lot of svgs on the page), and exceeds the timeout.

To resolve this simply increase the timeout by doing this.timeout(5000) in Mocha. I believe in Qunit you do QUnit.config.testTimeout = 5000.

In Mocha you can also set the timeout globally in test-helper.js:

test-helper.js

import resolver from './helpers/resolver'
import { setResolver } from 'ember-mocha'
import { mocha } from 'mocha'

mocha.setup({
  // customize as needed
  timeout: 5000
})
setResolver(resolver)

In describeComponent there is no this.timeout. So you can set the timeout in the beforeEach fucntion by doing

describeComponent(
  'frost-select',
  'Integration: FrostSelectComponent',
  {
    integration: true
  },
  function () {
    let props
    let dropDown
    beforeEach(function () {
      this.test._timeout = 6000
    })
    ...
})

Notes

Travis will upload the reports to Imgur

If you would like to help or have ideas on improving this tool I’m available on the Ember community slack @ewhite613 - issues and PRs also welcome :)