New Spring ’15 Feature: @testSetup

The Spring ’15 release is approaching quickly! As with every Salesforce release, it comes with a plethora of new features. Over the next few weeks, I am going to outline what I find to be some of the cooler features.

@testSetup

One of the nicest features of Spring ’15 in my mind is @testSetup. From the release notes:

Use test setup methods (methods that are annotated with @testSetup) to create test records once and then access them in every test method in the test class. Test setup methods can be time-saving when you need to create reference or prerequisite data for all test methods, or a common set of records that all test methods operate on.

Test setup methods can reduce test execution times especially when you’re working with many records. Test setup methods enable you to create common test data easily and efficiently. By setting up records once for the class, you don’t need to re-create records for each test method. Also, because the rollback of records that are created during test setup happens at the end of the execution of the entire class, the number of records that are rolled back is reduced. As a result, system resources are used more efficiently compared to creating those records and having them rolled back for each test method.

If a test class contains a test setup method, the testing framework executes the test setup method first, before any test method in the class. Records that are created in a test setup method are available to all test methods in the test class and are rolled back at the end of test class execution. If a test method changes those records, such as record field updates or record deletions, those changes are rolled back after each test method finishes execution. The next executing test method gets access to the original unmodified state of those records.

In Action

Let’s consider the following class:

public class AccountService {
    public static List<Account> getAllAccounts(){
        return [SELECT Id, Name FROM Account LIMIT 500];
    }
    
    public static List<Opportunity> getOpportunitiesOnAccount(Id accountId){
        return [SELECT Id, Name FROM Opportunity WHERE AccountId = :accountId];
    }
    
    public static void updateAccountName(Account account, String accountName){
        account.Name = accountName;
        update account;
    }
}

Typically, this would require a unit test class that looks like:

@isTest
public class AccountServiceTest {
    static testMethod void testGetAllAccounts(){
        List<Account> accounts = new List<Account>();
        for(Integer i=0;i < 10;i++){
            accounts.add(new Account(Name = 'Test Acc'));
        }
        insert accounts;
        
        Test.startTest();
        List<Account> accountsFromService = AccountService.getAllAccounts();
        Test.stopTest();
        
        System.assertEquals(10, accountsFromService.size(), 'There should be 10 accounts');
    }
    
    static testMethod void testGetOpportunitiesOnAccount(){
        Account acc = new Account();
        acc.Name = 'Test Account';
        insert acc;
        List<Opportunity> opportunities = new List<Opportunity>();
        for(Integer i=0;i < 10;i++){
            opportunities.add(new Opportunity(
                Name = 'Test Opp', 
                AccountId = acc.Id,
                StageName = 'Closed Won',
                CloseDate = Date.today()
            ));
        }
        insert opportunities;
        
        Test.startTest();
        List<Opportunity> opportunitiesFromService = AccountService.getOpportunitiesOnAccount(acc.Id);
        Test.stopTest();
        
        System.assertEquals(10, opportunitiesFromService.size(), 'There are 10 opportunities');
    }
    
    static testMethod void updateAccountName(){
        Account acc = new Account();
        acc.Name = 'Test Account';
        insert acc;
        
        Test.startTest();
        AccountService.updateAccountName(acc, 'Updated Account');
        Test.stopTest();
        
        Account accFromDb = [SELECT Id, Name FROM Account WHERE Id = :acc.Id];
        System.assertEquals('Updated Account', accFromDb.Name, 'The account name has been updated');
    }
}

So, what’s wrong with the current approach above? Well, you need to generate the same data several times, but it is also done in slightly different pieces of code. This can cause maintainability problems as well as well as a slow process generating the same data. How would this look using the new @testSetup method?

@isTest
public class AccountServiceTest {
    @testSetup static void setupTestData(){
        List<Account> accounts = new List<Account>();
        for(Integer i=0;i < 10;i++){
            accounts.add(new Account(Name = 'Test Acc'));
        }
        insert accounts;
        List<Opportunity> opportunities = new List<Opportunity>();
        for(Account acc:accounts){
            for(Integer i=0;i < 10;i++){
                opportunities.add(new Opportunity(
                    Name = 'Test Opp', 
                    AccountId = acc.Id,
                    StageName = 'Closed Won',
                    CloseDate = Date.today()
                ));
            }
        }
        insert opportunities;
    }
    
    static testMethod void testGetAllAccounts(){
        Test.startTest();
        List<Account> accountsFromService = AccountService.getAllAccounts();
        Test.stopTest();
        
        System.assertEquals(10, accountsFromService.size(), 'There should be 10 accounts');
    }
    
    static testMethod void testGetOpportunitiesOnAccount(){
        Account acc = [SELECT Id, Name FROM Account WHERE Name = 'Test Acc' LIMIT 1];
        
        Test.startTest();
        List<Opportunity> opportunitiesFromService = AccountService.getOpportunitiesOnAccount(acc.Id);
        Test.stopTest();
        
        System.assertEquals(10, opportunitiesFromService.size(), 'There are 10 opportunities');
    }
    
    static testMethod void updateAccountName(){
        Account acc = [SELECT Id, Name FROM Account WHERE Name = 'Test Acc' LIMIT 1];
        
        Test.startTest();
        AccountService.updateAccountName(acc, 'Updated Account');
        Test.stopTest();
        
        Account accFromDb = [SELECT Id, Name FROM Account WHERE Id = :acc.Id];
        System.assertEquals('Updated Account', accFromDb.Name, 'The account name has been updated');
    }
}

Note how there is only a single method that inserts data and also note that I can directly query records immediately in a test yet @SeeAllData is not set. We are accessing the data that was set up when the class was first instantiated. Now let’s tweak the code just slightly to verify the rollback works properly.

@isTest
public class AccountServiceTest {
    @testSetup static void setupTestData(){
        List<Account> accounts = new List<Account>();
        for(Integer i=0;i < 10;i++){
            accounts.add(new Account(Name = 'Test Acc'));
        }
        insert accounts;
        List<Opportunity> opportunities = new List<Opportunity>();
        for(Account acc:accounts){
            for(Integer i=0;i < 10;i++){
                opportunities.add(new Opportunity(
                    Name = 'Test Opp', 
                    AccountId = acc.Id,
                    StageName = 'Closed Won',
                    CloseDate = Date.today()
                ));
            }
        }
        insert opportunities;
    }
    
    static testMethod void testGetAllAccounts(){
        Test.startTest();
        List<Account> accountsFromService = AccountService.getAllAccounts();
        Test.stopTest();
        
        System.assertEquals(10, accountsFromService.size(), 'There should be 10 accounts');
    }
    
    static testMethod void testGetOpportunitiesOnAccount(){
        Account acc = [SELECT Id, Name FROM Account WHERE Name = 'Test Acc' LIMIT 1];
        
        Test.startTest();
        List<Opportunity> opportunitiesFromService = AccountService.getOpportunitiesOnAccount(acc.Id);
        Test.stopTest();
        
        System.assertEquals(10, opportunitiesFromService.size(), 'There are 10 opportunities');
    }
    
    static testMethod void updateAccountName(){
        Account acc = [SELECT Id, Name FROM Account WHERE Name = 'Test Acc' LIMIT 1];
        
        Test.startTest();
        AccountService.updateAccountName(acc, 'Updated Account');
        Test.stopTest();
        
        Account accFromDb = [SELECT Id, Name FROM Account WHERE Id = :acc.Id];
        System.assertEquals('Updated Account', accFromDb.Name, 'The account name has been updated');
    }
    
    static testMethod void testRollbackOnTestData(){
        List<Account> accounts = [SELECT Id, Name FROM Account WHERE Name = 'Updated Account'];
        
        System.assertEquals(0, accounts.size(), 'There are no accounts with that name because they have been rolled back');
    }
}

The last test was added to verify that the data rolls back to it’s original state, which it does (otherwise we would find an account with the updated name).

Other Considerations

From the release notes:

  • Test setup methods are supported only with the default data isolation mode for a test class. If the test class or a test method has access to organization data by using the @isTest(SeeAllData=true) annotation, test setup methods aren’t supported in this class. Because data isolation for tests is available for API versions 24.0 and later, test setup methods are also available for those versions only.
  • Multiple test setup methods are allowed in a test class, but the order in which they’re executed by the testing framework isn’t guaranteed.
  • If a fatal error occurs during the execution of a test setup method, such as an exception that’s caused by a DML operation or an assertion failure, the entire test class fails, and no further tests in the class are executed.
  • If a test setup method calls a non-test method of another class, no code coverage is calculated for the non-test method.

My Observations

This is a big step forward to achieving faster test runs. Faster unit test execution time is pivotal to Test Driven Development (TDD). This is a feature that will immediately impact the way everyone develops as every single Salesforce developer will want to utilize this functionality.

The main downside to all of this is the fact that the only way to access this data is to query it back from the database. While this may not be a problem in the simple example above, it may become more of an issue as the complexity grows.

With that said, it is still worth it to use this new functionality. You will still want to incorporate good unit test practices (such as using a TestUtils class), this will just be a new aspect of proper unit test structure. Enjoy and feel free to test it out on your own Spring ’15 Pre-Release Org!

26 Responses to “New Spring ’15 Feature: @testSetup”

  1. Rob
    January 27, 2015 at 3:14 pm #

    I agree this is a cool feature. I haven’t seen anything about how this plays with governor limits. Does what happens in a testSetup method count against the limits for every test in the class? Does it have its own context?

  2. Gareth Kavanagh
    January 27, 2015 at 5:41 pm #

    I’m delighted to see this feature, I’ve been writing setup classes at the end of my test classes for years so it looks like this could make that a bit easier. Thanks for the great article Jesse, much appreciated!

    • January 28, 2015 at 12:58 pm #

      Glad you liked it! Seems like a cool feature, I am looking forward to see how others utilize it as well.

  3. January 27, 2015 at 9:33 pm #

    Jesse,

    I agree that this is a good step in the right direction for TDD on the platform.

    One question that I have though is whether or not @testsetup methods could be configured in “test helper” classes and then somehow executed for each test class that needs access to that common test data. The use case that you mentioned above about having to setup the same data for multiple methods is what I am thinking about here but involving several test classes that all need the same initial data setup.

    Did you see any hints that suggest if this would be possible?

    • January 28, 2015 at 1:02 pm #

      No, I don’t believe that is possible. In this scenario, I would build a TestUtils class (annotated @isTest to avoid production use) and have a static method there that builds all of your data. At that point, it is just a matter of calling that single method in your @testSetup method for each class. It adds some redundant code, but makes it more maintainable.

      • January 29, 2015 at 10:56 am #

        This is exactly what I do. I have a testing utility class that builds data that is common or required for processing. Running the data setup in the new @testsetup will help reduce the duplicate setup code that is in multiple test methods.

  4. January 28, 2015 at 11:10 am #

    Thanks for the writeup, Jesse!

    Haven’t used @testsetup myself yet, but I have a couple questions that you may already have run into, that may make the difference about whether I’m able to use the feature myself:

    1. Do SOQL queries / DML statements, etc. in the @testsetup method count toward the Governor Limits of each testMethod in the Test Class, do they count toward Governor Limits at all? If @testSetup blocks had a separate “governor limits context”, that would be huge for testing very complex functionality.

    2. Is there a way to have the @testsetup method re-run before each Test Method in the class? Often you might want to have the same type of test data available before each Test Method is run, but you often want this data to be re-initialized before each Test Method is run… I can see in the example given above you in some cases you might want the effects of your Test Method runs to be “cumulative”, but I think that’s going to be the minority of cases, as it makes your test methods less modular and more interdependent, which makes them less manageable going forward, as different test methods have to be aware of the potential for other test methods to “mess with” the test data they’re looking for. So in these cases, we’ll probably have to stick with our traditional approach, of calling our own createTestData() method at the beginning of each test method, just to ensure that the test data is “fresh” for each method.

    • January 28, 2015 at 1:08 pm #

      Great questions!

      1) To be honest, I am not sure. I assume it might, but either way you probably want to have a Test.startTest() and a Test.stopTest() to make sure you only test the governor limits against the functionality being tested.

      2) I doesn’t appear that @testSetup is re-run before each test method. Instead, think of each method being run in the context of a single transaction. When your first test method is run, it rolls back your test data to the same exact state before it was run. This ensures each method starts with exactly the same data. It is not possible for one test method to affect another.

      • January 29, 2015 at 8:13 am #

        Just wanted to follow up on this Zach. Check the comment thread directly below this for an answer to #1. Long story short, yes they do use SOQL query limits.

  5. Marko Tomic
    January 29, 2015 at 3:26 am #

    Something is wrong with this feature and limits. It’s not only that limits are counted in the @testSetup (which is expected), they are staying after test.startTest() is called. For example this code produces too many SOQL queries 101:

    @isTest
    public with sharing class testtestsetup {
    @testSetup static void setupTestData(){
    for(integer i = 0; i < 100; i++){
    List accounts = [select id from Account];
    }
    }

    static testMethod void testGetAllAccounts(){
    test.starttest();
    List accounts = [select id from Account];
    test.stoptest();
    }
    }

    Regarding the feature, for me personally it’s totally useless because in all my tests I use heavily class global variables that are set in method for setting up test data. For some (to me) strange reason, global variables gets nullified after the @testSetup is finished so I have to query the data in every testMethod which actually makes more code instead of less. If I would use this new feature I would save one line of code at the beginning of every test method which is now call to method that setup test data but for every object that I created in this setup method I would need to query the DB. Not saying that I am wasting valuable SOQL queries on that.

    • January 29, 2015 at 7:59 am #

      This is very interesting. I just tested a very similar scenario and I was able to replicate your results. It appears that Test.startTest() and Test.stopTest() are not resetting the SOQL query limit. Interestingly enough, I did another test with DML statements and that didn’t fail. This leads me to believe that part of Test.startTest()/stopTest() is working, while some of it is broken. Here is the DML test I tried:


      @isTest
      public class AccountServiceTest {
      @testSetup static void setupTestData(){
      for(Integer i=0;i < 149;i++){ insert new Account(Name = 'Test Acc'); } } static testMethod void testAgainstLimits(){ Test.startTest(); for(Integer i=0;i < 30;i++){ insert new Account(Name = 'Test Acc'); } Test.stopTest(); System.assert(true, 'This was a success'); } }

      • January 29, 2015 at 8:43 am #

        There is a known issue related to the SOQL Queries, which doesn’t appear related to the @testSetup: “The Test.startTest() does not reset the number of SOQL queries” – https://success.salesforce.com/issues_view?id=a1p30000000T5R5AAK

        • Reid Carlberg
          January 29, 2015 at 10:06 am #

          I tested the work around in the known issue listing. It works here. Note the line I’ve added after the starttest() call.

          @isTest
          public with sharing class testtestsetup {
          @testSetup static void setupTestData(){
          for(integer i = 0; i < 100; i++){
          List accounts = [select id from Account limit 1];
          }
          }

          static testMethod void testGetAllAccounts(){
          test.starttest();
          Limits.getQueries();
          List accounts = [select id from Account];
          test.stoptest();
          }
          }

          • January 29, 2015 at 10:08 am #

            Awesome! Thanks for testing it Reid. That is good to know.

  6. Marko Tomic
    January 29, 2015 at 8:56 am #

    Thanks Peter,

    That explains a lot. I also was wandering why some of our production tests recently started to fail with to many SOQL queries. One would expect that such issue would be urgently fixed and deployed since it makes one of the toughest Salesforce limits even more restricted but it’s almost 3 months old without resolution 🙁

    • January 29, 2015 at 11:54 am #

      Hopefully it gets fixed. It is too bad that the static variables get reset after the @testSetup finishes execution. It would be very useful to be able to assign values created in @testSetup to static variables and then use them in the tests.

      Something like the following would be useful.

      private static Map<String, List> model;
      @testSetup static void testSetup() {
      model = generateTestData();
      }

      • Marko Tomic
        January 29, 2015 at 12:15 pm #

        Exactly, right on spot! It can be seen even in this Jesse’s example at the beginning of the test methods 2 and 3 where he reads account he has just created. It violates the DRY principle more then it supports with @testSetup because calling testSetup() in every test method costs only 1 line of code but reading objects created in testSetup for the purpose of assertion costs at least one line

        • February 10, 2015 at 11:13 am #

          You can still avoid repeating yourself, kind of. The first line of each @isTest method would be to call a static method retrieveTestDataModel(). That would do the querying.

          It would just be nicer to have all setup be able to be done as part of the @testSetup, instead of having it separated out into two different methods. That lends itself to errors in maintenance, because adding a new field, object, etc., requires an update to @testSetup and retrieveTestDataModel().

  7. Ron Hess
    March 3, 2015 at 11:17 pm #

    This issue has caused a failure to install my managed package, I had to back out all uses of @testSetup in any managed package or my customers could not install.

    It appears that the methods marked as @testSetup are always executed at Package Install time. This is not expected and blocks installs if the customer modified the objects ( as they sometimes do)

    i’ve filed a case with support.

  8. March 30, 2015 at 4:25 pm #

    Seems like a wonderful step in the right direction, hate that static variables are reset though.

  9. Lars Handrick
    June 9, 2015 at 5:49 am #

    What is the difference between @testSetup and a “static { /* init */ }” method? In my tests the restrictions above (seeAllTests and code coverage) didn’t work, so it seems this is better?

  10. August 29, 2017 at 3:05 pm #

    First off I would like to say superb blog! I had a quick question which I’d like
    to ask if you do not mind. I was curious to know how you center yourself and
    clear your mind prior to writing. I’ve had a hard time clearing my thoughts in getting my ideas out.

    I truly do enjoy writing but it just seems like the
    first 10 to 15 minutes are usually wasted simply just trying to figure
    out how to begin. Any ideas or hints? Thank you!

Trackbacks/Pingbacks

  1. How do governing limits work with @testSetup from Spring 15' release | DL-UAT - April 22, 2015

    […] new @testSetup method is covered pretty well by Jesse Altman in his: New Spring ’15 Feature: @testSetup article but as mentioned in the comments, there is a question about how this new feature works with the […]

Leave a Comment