Introduction

In the world of smart contracts, testing and validating their behaviour is crucial to ensure their proper functioning. Vault Core's API interface provides a powerful simulation capability for conducting such tests. In this article, we will dive into the Gherkin syntax, a plain-text language designed for non-programmers to create and understand test scenarios. By using Gherkin, smart contract engineers and business people can collaborate on creating comprehensive and readable test cases that target the requirements. We will explore the advantages and challenges of using Gherkin for smart contract simulation test-suites and give insight into how to get started.

Problem statement

Smart contracts in Thought Machines Vault Core act as the configuration layer dictating the behaviour of customer accounts. These contracts define the logic behind a financial product, responding to specific events as they arise. This provides banks with an unprecedented level of control over their product logic, empowering them to innovate and introduce new offerings to the market with greater agility.

One of the primary challenges and responsibilities of smart contract engineers is ensuring that smart contracts operate as intended. To achieve this, engineers adhere to a comprehensive automated testing strategy. Within this strategy, several automated testing approaches are crucial, including unit testing, integration testing, e2e testing, fast-forward and time-cursor testing, among others not mentioned here. Guided by the test-pyramid, this article specifically delves into the simulation testing technique, while highlighting its significance in the broader testing context.

Simulation testing is designed to verify contract behaviour. It involves running one or more smart contracts through the Vault Core API simulation interface. The result is an output that showcases the complete lifecycle of the account in a JSON stream, based on the given input parameters.

Despite its benefits, simulation testing presents challenges, especially in team collaboration. A recurring issue is the disconnect between technical and non-technical team members. This often leads to inefficiencies in test designs, as well as challenges in capturing and implementing the product owners' intended behaviour. Such discrepancies primarily stem from miscommunication, misunderstandings, and the absence of a standardised language that resonates with both technical and non-technical personnel. If unaddressed, these issues can lead to false contract implementations (bugs).

Another observed challenge pertains to the test-suite codebase. Less experienced engineering teams might grapple with maintaining a tidy and systematic codebase. This oversight can result in redundant tasks, inflated test-suites, and escalated test maintenance expenses, subsequently affecting development, deployment, essentially all key Dora metrics.

In summary, the realm of smart contract testing necessitates a refined approach. This enhanced strategy should champion efficiency, foster team collaboration, and bolster the overall quality and performance of smart contracts.

Smart Contract Simulation Testing

Vault Core offers an API interface for simulating Smart Contract and workflow behaviour. Simulation tests focus on sequences of events that represent business scenarios, or more complex integration scenarios. They are used to test contracts in their entirety in response to realistic inputs, such as receiving postings or the execution of schedules. The simulation framework simulates their behaviour in order to test the contract in isolation from the rest of Vault Core Services.

These tests will always require the same basic structure, which we’ll describe in phases:

1. Scenario set-up: where default parameters or data can be initialised. These activities may consist of pre-populating past events or historical data required by your object for testing.
2. Triggering the activity or action phase: that should generate the defined response from the system
3. Verification of results phase: usually done by making assertions on the response object from the previous step.

Great samples are available on the Vault Core docs hub in the Smart Contract Tutorial: https://docs.thoughtmachine.net/vault-core/4-6/EN/tutorials/smart-contracts/ This link requires Okta access - you can request it to be setup by ThoughtMachine.

A small example of a simulation test, for illustrative purposes only

1class DepositTest(unittest.TestCase):
2    @classmethod
3    def setUpClass(self):
4        self.client = ...
5        self.deposit_smart_contracts = ...
6
7    def test_regular_deposit(self):
8        start = datetime(year=2019, month=1, day=1, tzinfo=timezone.utc)
9        end = datetime(year=2019, month=1, day=2, tzinfo=timezone.utc)
10        regular_deposit_instructions = ...
11        
12        res = self.client.simulate_contracts(
13            start_timestamp=start,
14            end_timestamp=end,
15            self.deposit_smart_contracts,
16            regular_deposit_instructions,
17        )
18      
19        self.assertEqual(len(res[-1]["result"]["posting_instruction_batches"]), 1)

In this example, our scenario is described by the function name test_regular_deposit. We set up the parameters and conditions for the test in the first part. The behaviour we want to verify is triggered by the call to client.simulate_contracts. We then examine the result in the final segment with assertEqual.

But you’ll notice that unless you are technically inclined and familiar with software development, it may not be so easy to follow what is happening in the test. This means there’s no opportunity for non-technical people to directly review, provide feedback, and contribute to the test case.

Quick-start of using Gherkin for Simulation testing

Gherkin is a user-friendly plain-text language enriched with minimal structural elements, purposefully crafted for those without a coding background. This language offers a way to depict examples that showcase business rules across different real-world scenarios. One of its benefits is to highlight the purpose of any given test.

Non-technical Steps

If we apply Gherkin, we would start with composing the feature scenario description in a feature file deposits.feature.

1Feature: Deposits
2  Scenario: Unchallenged Deposit
3    Given a deposit product
4    When a deposit is made
5    Then there should be a successful posting 

In this basic example, we see that the steps start with the keywords Given, When, and Then.

From the Gherkin documentation:

Given steps are used to describe the initial context of the system - the scene of the scenario. It is typically something that happened in the past.

When Cucumber executes a Given step, it will configure the system to be in a well-defined state, such as creating and configuring objects or adding data to a test database.

When steps are used to describe an event, or an action. This can be a person interacting with the system, or it can be an event triggered by another system.

It's strongly recommended that you only have a single When step per scenario. If you feel compelled to add more, it's usually a sign that you should split the scenario up into multiple scenarios.

Then steps are used to describe an expected outcome, or result.
The step definition of a Then step should use an assertion to compare the actual outcome (what the system actually does) to the expected outcome (what the step says the system is supposed to do).

While many teams receive scenario descriptions in this style above, often they are shared in spreadsheets or on tickets only. Once the ticket description/text has been turned into test-code, there are two sources of truth for the test case, one in the ticket and one in the code. By using Gherkin and the following steps, we can marry the two into a single source of truth for the test case going forward also the non-technical people will refer to the Gherkin implementation on the repository to review the already implemented test scenarios.

Technical Steps

Developers can prepare the environment.py to run the simulation methods in the hooks.

1'''import ...
2
3# hooks
4def before_all(context):
5    context.failed_scenarios = [] 
6    
7# before every scenario
8def before_scenario(context, scenario):
9    '''reset context values
10
11# after every scenario
12def after_scenario(context, scenario):
13    try:
14        # build test object
15        simulation_test_case = SimulationTestCase()
16        
17        # run the test
18        simulation_test_case.run_test_scenario(
19          test_scenario = _get_simulation_test_scenario(context=context, scenario=scenario)
20        )
21    except Exception as e:
22        context.failed_scenarios.append(scenario)
23        logger.error(f'FAILED scenario "{context.name}"')
24        assert False, str(e)
25        
26def after_all(context):
27    if len(context.failed_scenarios) > 0:
28        fail(f'Total {len(context.failed_scenarios)} scenarios failed')
29
30# custom method to prepare the simulation test scenario
31def _get_simulation_test_scenario(context, scenario):
32    '''prepare ContractConfig object
33    contract_config = ...
34    return SimulationTestScenario(
35        start=context.simulation_start_date,
36        end=context.simulation_end_date,
37        sub_tests=context.sub_tests,
38        contract_config=contract_config,
39        internal_accounts=default_internal_accounts,
40        debug=debug,
41    )

The steps can then be added to a deposit_steps.py.

1@Given("a deposit product")
2def a_deposit_product(self):
3  extract_date(started_date)
4  extract_date(closed_date)
5  create_account(context, default_amount)
6  
7@When("a deposit is made")
8def a_deposit_is_made(self):
9  run_test_scenario(context)
10  
11@Then("there should be a successful posting")
12def there_should_be_a_successful_posting(self):
13  check_pib_with_timestamp(context)

The step files naturally separate code into logical components, helping keep our code DRY as these steps can be reused in other scenarios. Running the tests will then simply need to execute the following command:

1behave bdd_tests/features

where bdd_tests/features is the location for the feature files created.

There is obviously a little bit more effort in preparing tests using Gherkin syntax. However, as can be seen in our example, there are advantages to using it.

Advantages

• Collaboration and Readability: Gherkin scripts can be easily understood by business executives, developers, and non-programmers alike which enables better collaboration. The clear language ensures that tests are directly linked to business requirements, ensuring alignment with acceptance tests, a shared source of truth, which is self-documenting. You don’t need to be an expert to understand the concise Gherkin command set. The enablement for this close collaboration leads to fewer disconnects between requirement and implementation, less bugs.
• Efficiency in Writing and Reusing Code: The style of writing test cases in Gherkin makes it easier to reuse code in other tests, which in turn facilitates the development of new test scenarios based on previous step definitions.
• Consistency: Tests written in Gherkin maintain a uniform format, making it easier for team members to follow and contribute.
• Other Advantages: Gherkin is platform agnostic, which means tests can be run on different testing frameworks. It encourages behaviour-driven development, ensuring the smart contract behaves as intended.

Of course, there is no one perfect solution, and there are a few things to consider.

Challenges

• Test Construction and Reusability: Writing good tests can be challenging. Due to the flexibility of Gherkin, there's a risk of producing sloppy tests. Tests should be constructed to be reusable across various scenarios, and they should be clear to both technical and non-technical team members. Poorly written tests can inflate test-maintenance costs. Over time, tests that can't be reused or those that are resource-intensive can bloat the codebase and slow down the entire test suite, impacting development and deployment processes.
• Adopting a Behaviour-Driven Mindset: Transitioning from TDD to BDD is a paradigm shift. While TDD centres on code logic, BDD prioritises end-user behaviour.
• Collaboration and Business Engagement: Effective BDD requires high collaboration between product owners, business analysts, developers, and testers. While product owners and analysts draft specifications, developers and testers need to understand, implement, and sometimes suggest modifications. This collective effort ensures alignment between envisioned behaviour and implementation. It also requires a mindset shift on the part of non-technical people to be more involved with the actual implementation of the automated test-suite, rather than just handing over the requirements.
• Performance and Efficiency: Consistent behaviour independence can cause inefficiencies, notably if behaviours have alike setups. Simulation tests might have performance issues, making it vital to choose tests wisely. BDD isn't always ideal, especially for high-performance simulations; teams should assess its suitability.

Conclusion

In smart contract engineering, as in all other engineering, clear communication and teamwork are key. BDD with Gherkin offers a simple way to involve both technical and non-technical members in the testing process. It's important to remember, though, that it's not just about the tools we use, but also how we work together. At Ikigai Digital, we believe in using the best tools and methods, but we also value good communication and collaboration. Let's work together to make smart contract testing more efficient and clear for everyone, contact us to help introduce this technique for smart contract testing in your team.

References

• Gherkin Reference - Cucumber Documentation
• Thought Machine Vault Core - Smart Contracts