Containers Trend Report. Explore the current state of containers, containerization strategies, and modernizing architecture.
Securing Your Software Supply Chain with JFrog and Azure. Leave with a roadmap for keeping your company and customers safe.
The Testing, Tools, and Frameworks Zone encapsulates one of the final stages of the SDLC as it ensures that your application and/or environment is ready for deployment. From walking you through the tools and frameworks tailored to your specific development needs to leveraging testing practices to evaluate and verify that your product or application does what it is required to do, this Zone covers everything you need to set yourself up for success.
A system is a collection of interconnected components that work together to perform a defined function or set of functions. The components can be hardware, software, firmware, or a combination. In software, a system can refer to a collection of software modules, libraries, and frameworks that work together to achieve a specific goal. What Is System Testing? System testing is one type of software testing that involves testing the entire system as a whole to ensure that it meets the specified requirements and functions correctly. Systems testing is a critical phase of software development to ensure the system functions as expected and meets the specified requirements. System testing can be conducted in various ways, including manual testing, automated testing, or a combination of both. It involves testing the system at the integration and end-to-end levels to ensure that all the system components work together seamlessly. The main goal of system testing is to detect defects, errors, and inconsistencies in the system, including hardware, software, and other components. The following are some best practices for systems testing: Define Clear and Comprehensive Test Cases Ensure you understand the requirements and use cases for the system, and develop comprehensive test cases that cover all aspects of the system's functionality. Test cases should be well-defined and detailed and include all possible scenarios. Identify the requirements: The first step in defining clear and comprehensive test cases is to identify the requirements of the system or software tested. These requirements should be documented and agreed upon by all stakeholders. Define the scope: Once the requirements are identified, the scope of testing should be defined. This includes what functionalities will be tested, what data will be used, and what types of tests will be performed. Write test cases: You can start writing test cases based on the requirements and scope. Test cases should be written in clear, concise, and easy-to-understand language. Each test case should have a unique identifier, a summary of the test case, and steps to execute the test case. Include expected results: In addition to the steps to execute the test case, you should also include the expected results for each test case. This helps ensure that the test cases are comprehensive and cover all scenarios. Review and revise: Once the test cases are written, they should be reviewed and revised by a team of testers and stakeholders to ensure they are clear, comprehensive, and cover all requirements. Execute the test cases: Finally, the test cases should be executed, and the results should be documented. Any defects found during testing should be reported and tracked until they are resolved. Use Automated Testing Automated testing tools can be used to save time and reduce the potential for human error. A software testing technique involving specialized tools to execute test cases automatically is called automation testing without manual intervention. It is used to verify that the software meets its intended functionality, performance, and quality requirements. Here are some situations where automated testing can be helpful: Repetitive tests: Automated testing is ideal for tests that need to be executed repeatedly, such as regression testing, to save time and effort compared to manual testing. Large and complex systems: When a system is large and complex, manual testing can become impractical. Automated testing ensures that all system parts are working correctly. Performance testing: Automated testing tools can simulate multiple users to test the system's performance under various loads. Time-critical testing: Automated testing can run faster and provide immediate feedback, which is critical in time-sensitive projects. Regression testing: Automated testing is beneficial for regression testing, which involves verifying that new changes to the software have not affected existing functionality. Continuous integration/continuous delivery (CI/CD) pipelines: Automated testing is a crucial part of CI/CD pipelines, aiming to automate software development and release. Perform Tests Early and Often Begin testing as early as possible in the development cycle and continue testing throughout development. This approach will help identify defects early on, reducing the cost and time required to fix them. Utilize a Test Environment A dedicated test environment is necessary to simulate the production environment, including hardware, software, and data. Testing in a different environment helps to minimize the impact of errors and prevents interference with production systems. Conduct Thorough Performance Testing: Performance testing is critical to ensure the system can handle the expected load and usage. Tests should be conducted to measure the system's response times, resource utilization, and scalability under different loads. Ensure Compatibility Test the system's compatibility with different operating systems, hardware configurations, and other software that may interact with the system. Conduct Security Testing It is essential to ensure that the system is secure and that confidential data is protected. Security testing should include vulnerability scanning, penetration testing, Hardware security if the system is embedded system, and other security measures. Document Test Results Documenting test results, including issues found, helps track progress and ensure all defects are resolved. This documentation is helpful for future reference and can help identify trends and areas for improvement. Involve Stakeholders Stakeholders should be involved in the testing process, including end-users, developers, and management. This approach can help ensure that the system meets the expectations and requirements of all stakeholders. Final Verdict System testing is an essential process in the software development life cycle that ensures the system is ready for deployment and meets the end-users requirements. By guaranteeing these best practices are followed, you can ensure that the testing of your system is effective and efficient, leading to a successful project outcome.
If you’re working with smart contracts—or even just exploring them—you probably already know that smart contract security is important. Smart contracts are immutable once deployed, and often involve significant amounts of money. Writing safe and reliable code before deployment should be top of mind. And as the adoption of blockchain accelerates, ensuring the security of smart contracts becomes even more important. One of the best additions to your smart contract audit is fuzzing, a dynamic testing technique that exposes vulnerabilities by generating and injecting random inputs into your smart contracts during testing. In this article, we’ll explore how to use fuzzing to effectively audit a smart contract. Specifically, we’ll look at ConsenSys Diligence Fuzzing—a new fuzzing as a service (FaaS) offering. We’ll delve into the technical aspects and show some code examples. What Is Fuzzing? Fuzzing is a dynamic testing technique where random (or semi-random) inputs called “fuzz” are generated and injected into code. Fuzzing can help reveal bugs and vulnerabilities that weren’t caught by traditional testing methods. Manual (unit) testing requires you to figure out what functionality to test, what inputs to use, and what the expected output should be. It’s time-consuming, difficult, and in the end, it’s still easy to miss scenarios. On the other hand, fuzzing (or fuzz testing) is an automated testing process that sends random data into an application to test its security. A fuzzer can help you understand how a program responds to unpredictable inputs. Fuzzing has been around for a while. Defensics and Burp Suite are some examples in the traditional development world. There are also several Web3/blockchain fuzzing tools available, such as Echidna and Foundry. However, Diligence Fuzzing is fuzzing as a service and makes everything a little simpler to implement. Which in the end means better audits and more secure contracts. So let’s look into it in more detail. ConsenSys Diligence Fuzzing Diligence Fuzzing (by ConsenSys, which is also behind ecosystem standards such as MetaMask and Infura) is a fuzzer built for Web3 smart contracts. It: Works from a formal spec that describes the expected behavior of your smart contract Generates transaction sequences that might be able to violate your assertions Uses advanced analysis to find inputs that cover a maximum amount of your smart contract code Validates the business logic of the app and checks for functional correctness Provides you with any findings And all as a service with minimal work from you! To use Diligence Fuzzing follow these three steps: First, define your smart contract specs using Scribble. Next, submit the code to Diligence to run your fuzzing. Finally, with the audit report, fix and improve your code! Fuzzing in Action So let’s test it out and see it in action. We will use the Fuzzing CLI and Scribble to fuzz-test a sample smart contract. Step 1: Sign Up First, sign up for access to the Diligence Fuzzing. Step 2: Install Dependencies Next, install the Fuzzing CLI and Scribble. ConsenSys recommends that you have the latest versions of Node and Python. Be sure you are using at least Python 3.6 and Node 16. Then: pip3 install diligence-fuzzing npm i -g eth-scribble ganache truffle Note: This requires a Linux, mac, or Linux subsystem with Windows. Windows Powershell has some complexities the team is working on. You can always use a GitHub codespace (which creates a VScode-like-interface with a clean bootstrapped build) and install the above prerequisites via command line. Step 3: Get an API Key Now you need to generate an API key to use the CLI. Visit the API Keys page and click on Create new API Key. Step 4: Set Up Fuzzing Configuration Now we need a smart contract to fuzz! As part of their own tutorial, ConsenSys provides a sample smart contract to use. Let’s just use that one. git clone https://github.com/ConsenSys/scribble-exercise-1.git Open the .fuzz.yml file from the project and add in your API key for the “key” property at around line 25. # .fuzz_token.yml fuzz: # Tell the CLI where to find the compiled contracts and compilation artifacts build_directory: build/contracts # The following address is going to be the main target for the fuzzing campaign deployed_contract_address: "0xe78A0F7E598Cc8b0Bb87894B0F60dD2a88d6a8Ab" # We'll do fuzzing with 2 cores number_of_cores: 2 # Run the campaign for just 3 minutes. time_limit: 3m # Put the campaign in the Sribble Exercise 1 project project: "Scribble Exercise 1" # When the campaign is created it'll get a name <prefix>_<random_characters> campaign_name_prefix: "ERC20 campaign" # Point to your ganache node which holds the seed rpc_url: "http://localhost:8545" key: "INSERT YOUR API KEY HERE" # This is the contract that the campaign will show coverage for/ map issues to etc # It's a list of all the relevant contracts (don't worry about dependencies, we'll get those automatically ) targets: - "contracts/vulnerableERC20.sol" Note: Be sure to stop your fuzzing campaigns or set a time limit, or it might run for an unexpectedly long time. You’ll note from the above file that we set the time limit for our campaigns to three minutes. Step 5: Define Fuzzing Properties Notice also that we have our smart contract: contracts/vulnerableERC20.sol. Next, we need to define the properties we want the fuzzer to check in the smart contract. We’ll use Scribble for this step. Scribble is a specification language that translates high-level specs into Solidity code. It allows you to annotate your contracts with properties and then transforms those annotations into concrete assertions that can be used by testing tools (such as Diligence Fuzzing). Pretty cool! We will add the highlighted code segments to our contract: pragma solidity ^0.6.0; /// #invariant "balances are in sync" unchecked_sum(_balances) == _totalSupply; contract VulnerableToken { This annotation will ensure that our total supply and balances are in sync. Step 6: Run Now we fuzz! Simply run this command: make fuzz Step 7: Evaluate Results After the fuzzer is done (it might take a minute or two to start up) we can get our results. We can either use the link the fuzzer gives us, or we can go to our dashboard. Looking at properties, we can see what is being fuzzed and any violations. And guess what? We found a bug! Click on the line location button to see the offensive code. For details, click Show transaction details. We can see the fuzzer called “transfer”: Upon closer examination, we can now see what caused our bug. The transfer_to and origin arguments are the same. There must be a security vulnerability when someone sends tokens to themselves. Let’s look in the source code to see what’s wrong. function transfer(address _to, uint256 _value) external returns (bool) { address from = msg.sender; require(_value <= _balances[from]); uint256 newBalanceFrom = _balances[from] - _value; uint256 newBalanceTo = _balances[_to] + _value; _balances[from] = newBalanceFrom; _balances[_to] = newBalanceTo; emit Transfer(msg.sender, _to, _value); return true; } Yep! We can see that when the sender and recipient are the same, lines 30 and 31 will get a little weird—one is changing the value of the ‘from’ account, and one is changing the value of the ‘to’ account. The code assumes they are different accounts. But since they are the same account, by the time we get to line 31, the value we have is not the value we expect. It’s already been changed by the previous line. We can fix this by adding the highlighted lines of code below: function transfer(address _to, uint256 _value) external returns (bool) { address from = msg.sender; require(_value <= _balances[from]); _balances[from] -= _value; _balances[_to] += _value; uint256 newBalanceFrom = _balances[from] - _value; uint256 newBalanceTo = _balances[_to] + _value; _balances[from] = newBalanceFrom; _balances[_to] = newBalanceTo; emit Transfer(msg.sender, _to, _value); return true; } Here are several other technical details to be aware of: The seed.js script does some setup work for you here. It deploys the contract to a test node. It can also do things like mint tokens, open pools, etc. It gives the fuzzer the right state to start. The yml file has many config parameters that you can explore. Notably the contract address to fuzz, the API key, the time_limit for the fuzzing, and some others. The CLI ships with an auto-config generator. Run fuzz generate-config to get some helpful Q&A for generating the config. Smart Contract Audits—Use Fuzzing! Fuzzing and Diligence Fuzzing-as-a-service is a powerful tool for testing auditing Ethereum blockchain smart contracts. Whether you are working in decentralized finance (DeFi), NFTs, or just starting in smart contract development, it can take you to the next level of identifying and fixing vulnerabilities in your smart contracts. Along with manual reviews, unit tests, manual testing, penetration testing, code reviews, and more, fuzzing should be a key part of your smart contract security audit process for a more secure and robust codebase. Have a really great day!
I've been writing developer tests for a very long time. Lately, I've been reflecting on the types of tests I write and why some are easier than others. When teaching and coaching others how to write tests, I almost always explain what I mean by "Unit Tests" and "Integration Tests": Unit Tests don't touch hardware, don't do I/O, etc. (This should sound familiar to some of you, as it's the same idea as the set of unit testing rules by Michael Feathers written in 2005), and test against a single object or group of objects (I use the terms Sociable and Solitary to differentiate between the different kinds of "unit" tests, as defined by Jay Fields). I'd then demonstrate what I meant like this: Java @Test public void fullDeckHas52Cards() { Deck deck = new Deck(); assertThat(deck.size()) .isEqualTo(52); } @Test public void drawCardFromDeckReducesDeckSizeByOne() { Deck deck = new Deck(); deck.draw(); assertThat(deck.size()) .isEqualTo(51); } These are both "unit" tests (neither touch hardware nor do I/O) and are Solitary (the tests only reference the Deck class). Later on, we'd need to write tests against code that may do I/O, often a database or an external service (usually over the network). I'd explain that those were "Integration Tests" because we were integrating our code with someone else's code (the database code or the other service's code) through I/O. However, calling someone else's code that's supplied as a library (e.g., a JAR file) that doesn't do I/O (such as Caffeine, a caching library) can also be considered integration. Even using Java's Collection classes (e.g., ArrayList) is using someone else's code, though we don't usually think of that as integration. Hard To Redefine Terms With Lots of Baggage Over the past few years, I've become more frustrated with the terms "unit" and "integration" because: I have to explain what Unit and Integration mean before I can use them Everyone has their own internal definition of what they mean, which is often different from mine and other folks in the room Differences in definitions often led to long discussions that aren't useful Folks don't remember how I define it and fall back to their own definitions I don't find the debate over what "unit" means to be a useful exercise. I finally decided to do something about it and come up with different names. But first, I had to answer the question: why does it matter? What about Unit (doesn't do I/O) and Integration (may do I/O?) is important for the way I approach development? No I/O = Predictable and Fast If you create an object and call a method that only accesses internal properties (fields) and any parameters to the method, it must be predictable. Any logic or calculations the code is doing is deterministic. What makes code not predictable? I/O. Accessing a file is unpredictable because the file could've changed without you knowing, the drive could fail intermittently, could be out of space, etc. Accessing a remote service involves not only the network (unreliable) but also the remote service (unpredictable). I include access to anything outside of memory, such as random number generation and the current date and time as I/O, because they are also unpredictable. By eliminating all I/O, you make the code under test and, therefore, the test deterministic. Not accessing I/O also means your tests will run extremely fast (yes, there's code that performs lengthy calculations that run completely in memory, but I'm not talking about that), with most of the time spent getting the tests ready to run: compiling, starting, etc. Everything else, from instantiating objects to running the code, is almost instantaneous. There's no reason you can't have many thousands of tests run in a few seconds. By the way, this speed is critical for doing test-driven development, which is why I focus on these kinds of tests. Once set up, running tests without I/O is almost instantaneous. Now when it comes to code that does interact with the outside world, such as getting the current date and time or fetching information from a database or external service, I still want tests that don't do I/O. This is where Test Doubles come into play. In Hexagonal Architecture, that might mean a Stub or a Fake in place of a concrete Adapter, or I might use Nullable Infrastructure Wrappers, which is Stub-like implementation embedded in your production code. The idea is that we're still not touching I/O, so everything is still Predictable and Fast. These are a form of Sociable tests, where we're testing a larger set of collaborating classes but explicitly not testing the I/O itself. I want 80-90% of my tests to be these kinds of tests. Of course, it'll vary widely depending on how much your system is doing things vs. integrating with other things. I/O = Unpredictable and Slower At some point, though, you want to have some sort of test that touches I/O. Things like: Check the database schema is valid Ensure the ORM can read and write from the database and create well-formed objects Call an external service API and ensure you get a valid response These are hard to write because, as I said, they're unpredictable. Databases are often managed ("owned/defined") by the application, so those can use tools like Docker and Testcontainers to allow your tests to use real databases. They're slower, but you won't be running them nearly as often as the other tests. When it comes to calling external services, if it's a service that you can run inside a container (maybe it's Kafka, or it's a custom service created by another team that provides an executable), then you can do the same thing as with the database above. But if it's a public service (like GitHub or Google), or any service that you can't run locally or in a container, you're not going to be able to run automated tests that are predictable. Yes, you could run against a "sandbox" environment, but for me, that falls under "unpredictable" based on my experiences (maybe yours work better?). So, if I write these tests, I run them manually or have other ways of checking that my code works (like good monitoring and observability in production). Naming Attempts Naming is hard, as we all know. Trying to find a name that is descriptive and memorable but with little or no baggage is quite a task. First Attempt During a discussion of the problem with Willem Larsen, he proposed using words from a different language (e.g., Japanese) for the same terms (or at least what they meant to me). However, not being able to speak Japanese, I had to rely on less-than-perfect translation systems. Not only did I want to find a word or phrase that had the intent I wanted, but it also had to be relatively short. For months, I played with different translations but wasn't happy with the results, so I shelved the idea. Pure and Impure In Functional Programming, the terms "Pure" and "Impure" have somewhat similar meanings to my usage of "Unit" and "Integration." In FP, a pure function is a function that doesn't cause any side effects and always produces the same output for a given input. I tried using this terminology for a while, but it had two problems: I still had to define "pure" as not accessing I/O I got objections from FP folks because of the misuse of the term "pure" in the context of testing stateful object-oriented code (methods accessing internal state could never be "pure") Honestly, #1 was the more annoying aspect for me, though it did have the benefit of not coming with much baggage (except for folks familiar with FP). I do use the term "purify" when I'm talking about the process of separating the I/O code from the logic, e.g., "Let’s purify this code, making it I/O-Free." I/O-Free and I/O-Dependent So, here we are. The two main kinds of tests I write. The majority are I/O-Free tests, and when I hit the I/O boundary and need to test against some real external service, I use I/O-Dependent tests (or I/O-Based). I further split I/O-Free tests into Domain, Application, and other buckets, depending on the application architecture (e.g., Domain tests never have Test Doubles), but that's an article for another day.
Smart contract development, more so than most web2 development, requires thorough testing and careful deployment. Because smart contracts are immutable and often involve large sums of money, it’s very important to do all you can to be sure they are secure, reliable, and as free from bugs as possible. Two important tools for achieving these goals are the same tools most web3 devs use for their day-to-day work—Truffle and Infura. These tools give you what you need to write, test, debug, and deploy your smart contracts. In this article, we’ll walk you through a step-by-step guide on how Truffle and Infura can be utilized to debug and deploy smart contracts on Ethereum. We’ll begin by establishing a development environment and creating a basic smart contract using Solidity. Next, we’ll conduct debugging of the contract, followed by deployment using Infura, and ultimately debugging on the blockchain. Finally, we will provide some recommendations for writing resilient and secure smart contracts. Let's get started! Setting up the Development Environment Our first step is to establish a basic smart contract development setup using Truffle and Infura. However, it is important to note that this is not a comprehensive guide for setting up the environment, as there are already numerous tutorials available for this purpose. If you need a little background on smart contracts, blockchain, Ethereum, etc., then check out my previous article: Learn To Become a Web3 Developer by Exploring the Web3 Stack. Prerequisites Node.js: Make sure you have Node.js installed on your system. You can download and install the latest version from the official website. Truffle: Truffle is a suite of development tools for smart contract development. It gives you the tools you need to manage your workflow, test, deploy, run local blockchains, and more. You can install it globally using npm, the Node.js package manager. Run the following command in your terminal: Infura: Infura is a set of blockchain APIs that provides a simple and reliable way to connect to the Ethereum (and others) network without running a full node. It’s the industry standard way to access blockchains etc. You'll need to create an account on the Infura website and obtain an API key to use their services. Configuring Truffle Once you've installed Node.js and Truffle, you're ready to create a new Truffle project. Simply run the following command in your terminal to create a new project: This command will create a basic Truffle project with everything you need to get started on a new project. The contracts/ folder is where we'll write our smart contract, while the migrations/ folder is where we'll write migration scripts to deploy our contract to the blockchain. Next, we need to configure Truffle to use our local blockchain simulator. Open the truffle-config.js file and modify the development network to point to Ganache. Ganache allows you to fire up a personal instance of Ethereum for local development and testing. Here's an example configuration: JavaScript module.exports = { networks: { development: { host: "127.0.0.1", port: 8545, network_id: "*" } } }; Debugging Smart Contracts Locally Now that we have set up the development environment let's start by writing a simple Ethereum smart contract using Solidity. Smart contract development is a big topic! For an intro, check out this 10-minute orientation. For the purpose of this tutorial, we’ll keep it super simple and create a contract that simply allows users to store and retrieve a string value. JavaScript // SimpleStorage.sol pragma solidity ^0.8.0; contract SimpleStorage { string private value; function setValue(string memory _value) public { value = _value; } function getValue() public view returns (string memory) { return value; } } Save the source code above into a file named SimpleStorage.sol. The Truffle Debugger offers two ways to work with this code in your local environment. These are in-test debugging and read-only debugging calls. In-test debugging works within tests and is quite simple. You just wrap the line of interest in a debug statement like this. JavaScript it("should get latest result", async function() { const result = await debug( SimpleStorage.getValue() ); }); When you run your tests, add a debug flag and see the magic happen. This will pause the tests at the designated debug line and then launch the debugger's CLI interface, which allows you to step through code, inspect variables, and set breakpoints. Another way to access the Truffle Debugger is through read-only debugging calls. This method is preferred by many developers because it uses a transaction hash to access the debugger. With read-only debugging, developers can debug a transaction that has already been executed on the blockchain, making it a more practical method for debugging. This will open the Truffle Debugger in the terminal and then allow you to step. For an exhaustive list of options available in debug mode, check out the documentation. Deploying Smart Contracts With Infura Once you've written and tested your smart contract locally, it's time to deploy it. While it’s possible to deploy a contract using your own node, this can be time-consuming and resource-intensive. An alternative approach is to use a remote node like Infura. Infura is a node provider that gives you a simple and reliable way to deploy smart contracts to various layer 1 blockchain networks. To use Infura for contract deployment, you'll need to sign up for an account and create a new project (which you probably did above). Once you've done that, you'll be able to access your project's API endpoint, which you'll need to use to interact with the blockchain network. This tutorial offers a step-by-step guide if you need help with this. To deploy our smart contract to the Ethereum network, we'll need to connect to Infura using our API key. Open the truffle-config.js file again and add the following configuration for the Sepolia network. (Replace <PROJECT_ID> with your Infura project ID.): JavaScript const HDWalletProvider = require("@truffle/hdwallet-provider"); const infuraProjectId = "<PROJECT_ID>"; const mnemonic = "your mnemonic goes here"; module.exports = { networks: { development: { // ... }, sepolia: { provider: () => new HDWalletProvider(mnemonic, `https://sepolia.infura.io/v3/${infuraProjectId}`), network_id: 11155111, gas: 4000000 } } }; Note that we're using the @truffle/hdwallet-provider package to connect to Infura with our mnemonic. You can replace Sepolia with the name of the Ethereum network you want to deploy to. JavaScript truffle migrate --network sepolia We’re using a test Ethereum network here because there is a cost (gas fee) associated with deploying smart contracts. To pay this fee, you need ETH. On the Ethereum main net, you have to buy ETH—and the expenses can add up quickly. But on the test networks, there are faucets where you can get test ETH for free. Debugging Online Smart Contracts While debugging smart contracts locally will be the most common task, there may be occasions when you need to debug a contract that has already been deployed for some reason. Fortunately, the Truffle Debugger can accommodate this scenario as well. The added benefit is that it doesn't necessarily have to be your own contract as long as it has been verified on the blockchain. To debug a smart contract on a deployed chain, you can use the Truffle Debugger and Infura just as you did for local debugging. The only difference is that you'll need to specify the network to connect to. Here's an example of how to do it: In this example, we're connecting to the Sepolia testnet using the --network flag and specifying the transaction hash of the contract we want to debug. Once connected, you can use the same debugging techniques as you did for local debugging, such as stepping through code, inspecting variables, and setting breakpoints. Just keep in mind that network conditions may be different than your local machine, so be sure to thoroughly test your smart contract before deploying to an online chain. If you are still not feeling confident enough, you can take a look at this nice video tutorial. Best Practices for Testing and Deployment As you continue to develop and deploy smart contracts, here are some basic best practices to ensure your code is reliable and secure. Most of these are best practices for any type of coding—so be sure to keep them in mind: Test your smart contracts extensively before deploying them: Before deploying your smart contracts, make sure you test them thoroughly to catch any bugs or errors. Use the Truffle Debugger to debug your code locally (and on a deployed chain if necessary) to make sure it behaves as intended. Smart contracts on Ethereum are immutable. Be careful with production deployments. Use meaningful variable names: Give your variables meaningful names so that it's easier to debug your code. Avoid using abbreviations and acronyms unless they are widely understood. Comment your code: Like with all code, add comments to explain what it does and why it's necessary! This will help you and other developers understand the code and make changes to it later. Keep your contracts simple: Complex smart contracts can be difficult to test and debug. Whenever possible, break your contracts into smaller, more manageable pieces. Consider gas costs: Make sure you optimize your code to reduce gas costs and keep them as low as possible. Gas prices can drive up your expenses quickly. Keep your dependencies up to date: Make sure you keep your dependencies up to date to avoid any security vulnerabilities. Use tools like Truffle's truffle-verify plugin to verify your smart contracts and ensure they are secure. To dive in further, check out these best practices more specific to smart contracts. Conclusion We explored the benefits of using the Truffle Debugger and Infura in tandem to enhance the security and reliability of smart contracts. We have also seen how to deploy smart contracts on an EVM-based network using Infura and then how to debug them on a deployed chain. This is just the beginning! Once you are familiar with tools such as Truffle and Infura, you’re ready to start exploring smart contracts and blockchain development. Have fun!
Integration flows often interact with multiple external services such as databases, MQ queue managers, CICS regions, etc., and testing the flows has historically required all of the services to be available when running tests. This provides a high degree of confidence that the flows behave correctly for the tested scenarios, but the number of scenarios that can be tested this way is often too small to provide sufficient confidence that the overall solution will behave correctly in all (or even most) circumstances. Unit testing with mocked services is a common solution to this problem in the application development world, but integration solutions may require an in-between style of testing due to the large number of service interactions and the common re-use patterns seen in the integration world. App Connect Enterprise development started calling these “component tests” some time ago: Unit tests that test slightly larger sections of code and (unlike pure unit tests) are allowed to communicate with external services. This article will attempt to illustrate this approach using a database as an example service. Motivation The following flow is an example of the style of flow that might benefit from component testing: The flow accepts HTTP input, validates it, calls a stored procedure in the database, augments the results using another database table, formats the reply, and sends it back over HTTP. Errors must be handled in various places (which adds to the test requirements) and it is quite likely that some or all of the flow is shared with other flows as part of a library of code in order to aid flow development across the organization. This flow can be tested end-to-end using HTTP-based test tools, and it is clearly a good idea to have some verification that the flow does actually work when connected to live services. However, unit testing provides much faster feedback, does not require a copy of the database to be accessible, and can be run without needing external test tools, so one possible approach is to use a “mock database” instead of a real one, and create unit tests for the various parts: This approach is much faster to run (potentially runs in a few seconds), can be applied to library code, and the unit tests can be run anywhere (including developer laptops and in build pipelines), so it works very well as long as the database definitions are not changed by other parties. If the database definitions are changed, then the mocks become out of date, and the tests will pass even though the flows will fail when deployed. This is quite a common problem, as the integration developers are rarely able to guarantee the stability of the services being called by the flows, and so an additional set of tests is needed. These should use very similar technology to unit tests (to allow re-use of skills) so they can be quick to write and run, but should avoid using mocks: The term “component test” was used in the ACE product development pipeline for this sort of test to mean “unit tests that use external services,” and is distinct from integration testing because component tests only focus on one service: If a flow uses a database and CICS, then there would be separate tests for database and CICS interactions, with the tests only relying on one service. The name “component test” is unfortunately over-used in the software world, but does at least provide clarity if used consistently. Test Design Principles This conceptual split allows some useful design principles, summarized as follows: Unit tests should test flow code (flow structure, ESQL, Java, etc.) and must be able to run anywhere without requiring live links to actual services. These tests are fast and can be run on developer laptops, and are (when automated) one of the best ways to ensure the code does what the developer wrote it to do, both at the time of implementation and in the future. Having no service interactions also eliminates the problem of having to remember which unit tests need services and which don’t — no need to think, “Some unit tests run anywhere but others do not, so is my unit test failure a real code problem, or have I just forgotten some setup?” Component tests are similar to unit tests and are written using unit test frameworks, but are allowed to invoke external services, and as a result, these might not run on a developer laptop without some setup and/or service provisioning. Because they focus on one piece of a solution (a common library, for example), they can catch defects in that piece efficiently, but were hard to construct before ACE v12 introduced JUnit support for flow testing. Integration tests are those where the full set of services are used, covering the interactions between inputs and outputs of multiple services along with flow logic and code. These tests are often driven via external interfaces (REST APIs, queues, etc.) to provide whole-system testing, and connect to many different systems. These tests are essential, but using them to test the internal flow code in detail is similar to x-ray crystallography — firing inputs at the flow from the outside, and trying to deduce the internal code characteristics from the resulting outputs. Unit and component tests provide much better support for querying internal state, and provide complementary test coverage. Pipeline View When the different tests are included in a build pipeline, the result would be similar to the following: The unit tests are expected to be run on developer laptops, but are also run in the pipeline to ensure the code works as expected when multiple developers merge code at the same time or when the server code level is upgraded (among other reasons). Component tests might be run by developers, but are also run in the pipeline to test as many of the external service interactions as possible, and then integration tests cover the rest before the packaged solution is deployed to the next stage (which may include manual testing). Historical Perspective In the past, most testing of integration flows would have started on a developer laptop (usually with links to real services) before the pipeline is launched, and then continued in the integration testing phase with all services available; there was little automated unit or component testing. One reason for the emphasis on integration testing is that it closely mirrors how the real flows will run: In many cases, the whole point of the flows is that they connect systems together, so it makes sense to ensure that they actually do their job when given real services. Another reason for favoring integration testing is the historic difficulty in constructing automated testing for smaller components of flows (subflows, individual nodes, JavaCompute or ESQL methods, etc.) in older product releases. While scaffolding flows could be used to achieve this along with careful code construction, the initial cost and ongoing maintenance of the test structures meant that this was not a common practice. Despite (and sometimes because of) these reasons, integration testing tends to be slower, and due to the large amount of code being tested it tends to be harder to isolate bugs found at this stage; shifting testing further left to component tests (or unit tests if possible) normally leads to faster feedback when defects are introduced into the code, allowing for shorter development cycles while maintaining quality. Summary The term “component test” is not new, but the distinction between that and unit testing is hopefully helpful in explaining what integration developers can expect to run on their laptops without setup, and which tests might be expected to cover different functionality. See also https://github.com/ot4i/ace-demo-pipeline for an example of unit and component tests in a pipeline, using industry-standard pipeline technology to show how to integrate testing into ACE development.
Data quality testing is the process of validating that key characteristics of a dataset match what is anticipated prior to its consumption. According to Gartner, bad data costs organizations on average an estimated $12.9 million per year. In fact, Monte Carlo’s own research found that data engineers spend as much as 40% of their workday firefighting bad data. Those are some big numbers. Data quality issues are some of the most pernicious challenges facing modern data teams, and testing is one of the very first steps a data team will take on their journey to reliable data. Whether by mistake or entropy, anomalies are bound to occur as your data moves through your production pipelines. In this post, we’ll look at 7 essential data quality tests you need right now to validate your data, plus some of the ways you can apply data quality testing today to start building out your data quality motion. So, What Is Data Quality Testing Anyway? When it comes to data engineering, quality issues are a fact of life. Like all software and data applications, ETL/ELT systems are prone to failure from time to time. Among other factors, data pipelines are reliable if: The data is current, accurate, and complete. The data is unique and free from duplicates. The model is sound and represents reality. The transformed data is free from anomalies. While there’s no silver bullet for data quality issues, data quality testing empowers engineers to anticipate specific, known problems and write logic to proactively detect quality issues before they can impact downstream users. So, now that we have a common understanding of what data quality testing is, let’s look at some of the most common data quality tests you can run right now and how they might be used to quality detect issues in your data. NULL Values Test One of the most common data quality issues will arise from missing data, also known as NULL values. (Oh, those pesky NULL values). NULL values occur when a field is left blank, either intentionally or through a pipeline error, such as those caused by an API outage. Let’s say you were querying the impact of a marketing program on sales lift by region, but the ‘region’ field was left blank on multiple records. Any rows where ‘region’ was missing would necessarily be excluded from your report, leading to inefficient future spend on regional marketing programs. Not cool. As the name implies, a NULL values test will validate whether values within a specified column for a particular model are missing after the model runs. One excellent out-of-the-box test for uncovering NULL values is dbt’s generic not_null test. tests/test_not_null.sql {% test not_null(model, column_name) %} select * from {{ model } where {{ column_name } is null {% endtest %} Volume Tests Is data coming in? Is it too little? Too much? These are all data quality issues related to the volume of data entering your database. Volume tests are a must-have quality check that can be used to validate the number of rows contained in critical tables. Missing Data Let’s say your data platform processes data from temperature sensors, and one of those sensors fails. What happens? You may get a bunch of crazy temperature values—or you may get nothing at all. Missing data can quickly skew a data model or dashboard, so it’s important for your data quality testing program to identify quickly when data volume has changed due to missing data. Volume tests will enable you to identify when data volumes have changed to uncover failure points and validate the accuracy of your data. Too Much Data Too much data might not sound like a problem (it is called big data, after all), but when rows populate out of proportion, it can slow model performance and increase compute costs. Monitoring data volume increases can help reduce costs and maintain the integrity of models by leveraging only clean high-quality data that will drive impact for downstream users. Volume SLIs SLAs (service level agreements) are critical to a modern data reliability motion, and volume SLIs are the metrics that measure performance against a given SLA. Volume tests can be used to measure SLIs by either monitoring table size or table growth relative to previous measurements. For example, if you were measuring absolute table size, you would trigger an event when: The current total size (bytes or rows) decreases to a specific volume The current total size remains the same for a specific amount of time Numeric Distribution Tests Is my data within an accepted range? Are my values in range within a given column? These are questions that can be answered using distribution tests. In academic terms, a distribution test validates whether the data within a given table is representative of a normally distributed population. Essentially, does this data reflect reality? These rules can easily be created in SQL by defining minimums and maximums for a given column. One great example of an out-of-the-box distribution test is dbt’s accepted_values test, which allows the creator to define a range of acceptable distribution values for a given column. Great Expectations also provides a library of common “unit tests," which can be adapted for your distribution data quality testing. For example, here’s how you might ensure the zip_code column represents a valid zip code using Great Expectations unit tests: expect_column_values_to_be_between( column="zip_code", min_value=1, max_value=99999 ) Inaccurate Data In the same way that missing data can paint a false picture of reality, inaccurate data can be equally detrimental to data models. Inaccurate data refers to the distribution issues that arise from incorrectly represented datasets. Inaccurate data could be as simple as a doctor mistyping a patient’s weight or an SDR adding an extra zero to a revenue number. Creating distribution monitors to identify inaccurate data is particularly important for industries with robust regulatory needs, like healthcare and financial institutions. Data Variety Sometimes new values enter a table that fall outside a typical distribution. These values aren’t necessarily anomalous, but they could be. And when it comes to data quality, “could be” is usually something we want to keep an eye on. Including distribution tests as part of your data quality testing effort can help proactively monitor for new and unique values to spot potentially anomalous data that could indicate bigger issues down the road. Uniqueness Tests Another common quality issue that beleaguers data engineers and downstream users alike is duplicate data. Duplicate data is any data record that’s been copied and shared into another data record in your database. Without proper data quality testing, duplicate data can wreak all kinds of havoc—from spamming leads and degrading personalization programs to needlessly driving up database costs and causing reputational damage (for instance, duplicate social security numbers or other user IDs). Duplicate data can occur for a variety of reasons, from loose data aggregation processes to human typing errors—but it occurs most often when transferring data between systems. Uniqueness tests enable data teams to programmatically identify duplicate records to clean and normalize raw data before entering the production warehouse. If you’re using dbt, you can use the unique test to validate your data for duplicate records, but uniqueness tests are available easily out-of-the-box for a variety of tools depending on what you’ve integrated with your data stack. Referential Integrity Tests Referential integrity refers to the parent-child relationship between tables in a database. Also known as the primary key and the foreign key, the primary key is the root data that gets joined across tables to create models and derive insights. But what happens if the data used for that primary key gets changed or deleted? That’s where referential integrity tests come in. Known as the relationships test in dbt, referential integrity tests ensure that any data reflected in a child table has a corresponding parent table. Let’s say your marketing team is pulling a list of customer IDs to create a personalization campaign for the holidays. How does your marketing team know those customer IDs map back to real people with names and addresses? Referential integrity data quality testing ensures that no changes can be made to a parent or primary key without sharing those same changes across dependent tables. String Patterns In today’s distributed world, it’s not uncommon to find discrepancies in your data from all kinds of human error. Maybe a prospect forgot a character in their email address or an analyst accidentally changed a row without realizing it. Who knows! Because data inconsistencies are fairly common, it’s important that those records are reconciled regularly via data quality testing to ensure that your data stays clean and accurate. Utilizing a string-searching algorithm like RegEx is an excellent way to validate that strings in a column match a particular pattern. String patterns can be used to validate a variety of common patterns like UUIDs, phone numbers, emails, numbers, escape characters, dates, etc. Freshness Checks All data definitely has a shelf life. When data is being refreshed at a regular cadence, the data paints an accurate picture of the data source. But when data becomes stale or outdated, it ceases to be reliable, and therefore, useful for downstream consumers. And you can always count on your downstream consumers to notice when their dashboards aren’t being refreshed. Freshness checks validate the quality of data within a table by monitoring how frequently that data is updated against predefined latency rules, such as when you expect an ingestion job to load on any given day. Freshness tests can be created manually using SQL rules, or natively within certain ETL tools like the dbt source freshness command. Freshness SLIs In the same way you would write a SQL rule to verify volume SLIs, you can create a SQL rule to verify the freshness of your data. In this case, the SLI would be something like “hours since dataset refreshed.” Like your SLI for volume, what threshold you choose for your SQL rule will depend on the normal cadence of your batched (or streamed) data and what agreements you’ve made in your freshness SLA. Testing Is Time Intensive Now that you have a few essential data quality tests in your data quality testing quiver, it’s time to get at it! But remember, data reliability is a journey. And data quality testing is just the first step on your path. As your company’s data needs grow, your manual data quality testing program will likely struggle to keep up. And even the most comprehensive data quality testing program won’t be able to account for every possible issue. If data quality testing can cover what you know might happen to your data, we need a way to monitor and alert for what we don’t know might happen to your data (our unknown unknowns). Data quality issues that can be easily predicted: For these known unknowns, automated data quality testing and manual threshold setting should cover your bases. Data quality issues that cannot be easily predicted: These are your unknown unknowns. And as data pipelines become increasingly complex, this number will only grow. Still, even with automated data quality testing, there’s extensive lift required to continue updating existing tests and thresholds, writing new ones, and deprecating old ones as your data ecosystem grows and data evolves. Over time, this process becomes tedious, time-consuming, and results in more technical debt that you’ll need to pay down later. That's why leveraging machine learning to detect data anomalies is so powerful. It accounts for unknown unknowns and can scale with your data environment. If data reliability is on your roadmap next year, consider going beyond data quality testing and embracing a more automated, scaled approach to data quality.
With cy.intercept(), you can intercept HTTP requests and responses in your tests, and perform actions like modifying the response, delaying the response, or returning a custom response. When a request is intercepted by cy.intercept() the request is prevented from being sent to the server, and instead, Cypress will respond with the mock data you provide. This allows you to test different scenarios and responses from a server without actually having to make requests to it. Before intercepting network requests, one of the main challenges was that it was difficult to debug and diagnose network-related issues. Developers needed more visibility into what was happening with network traffic between a client and a server. Intercepting network requests provides insight into the network traffic generated by the application. Without this capability, troubleshooting issues can become more complex and time-consuming. The team may not have the necessary information to identify the cause of the problem, which can result in delays in the testing process. Moreover, QA teams had little access to the requests and responses transmitted between the client and server due to their inability to intercept and examine network data. Because of this, it was challenging to comprehend the application’s behavior. Many tools can be used for intercepting network requests. Cypress is one of the most popular automation testing frameworks through which you can intercept network requests. Cypress intercept — cy.intercept() is a method provided by Cypress that allows you to intercept and modify network requests made by your application. It will enable you to simulate different server responses or network conditions to test how your application handles them. This can be very useful when writing end-to-end tests. This can be very useful when writing end-to-end tests. To use the Cypress intercept — cy.intercept() method, you can call it within a Cypress test like this: cy.intercept(<url>, <options>) The parameter specifies the URL of the network request you want to intercept and is an object that can be used to specify additional options, such as the response to return and the status code to use, and so on. Before deep diving into using Cypress intercept for handling network requests, let’s first understand the nuances of network testing while performing Cypress testing. What Are Network Requests? Network requests refer to exchanging data between a client and a server over a network. When testing web applications, verifying the correct metadata, such as headers, cookies, authentication tokens, and other information sent with the request, can be important. In testing, network request metadata can be used to verify that the correct headers and cookies are being sent with the request, to ensure that the data is being sent in the correct format, and to verify that the authentication tokens are being set correctly. Tools such as Cypress provide APIs for intercepting and inspecting network request metadata, making it easy to test the behavior of network requests and ensure that they are being handled correctly. Network requests can be made using HTTP (Hypertext Transfer Protocol) or HTTPS (Hypertext Transfer Protocol Secure) protocols. HTTP is the traditional protocol for sending and receiving data over the Internet. It’s a simple, text-based protocol for sending and receiving information between clients and servers. HTTPS is a secure version of the HTTP protocol that uses SSL/TLS encryption to secure the data transmitted between the client and the server. HTTPS protects sensitive data, such as passwords, credit card information, and other personal information. When you make an HTTPS request, your browser establishes an encrypted connection with the server. All data exchanged between the two parties is encrypted to ensure that third parties cannot intercept and read it. This is crucial to how the Internet works and allows data transfer between devices. Here’s an example: A user opens a web browser and types in the URL for a website, such as “www.example.com.” The browser sends a network request to the server hosting the website, asking for the HTML, CSS, and JavaScript files that make up the website. The server receives the request and sends back the requested files. The browser receives the files and uses them to render the website on the user’s screen. The user interacts with the website by clicking a button or filling out a form. The browser sends another network request to the server with additional data, such as the information the user entered in the form. The server processes the request and returns the appropriate response, such as confirming the form submission or sending back data for the next page of the website. In the below diagram, you can see there is a network request between a client (the web browser) and a server (the website host). The client requests resources, and the server sends back the requested information. The process can be repeated as the user interacts with the website. What Is Intercepting Network Requests? Intercepting network requests refers to intercepting and inspecting the traffic between a client and a server during communication over a network. Web development usually involves intercepting HTTP requests and responses. Intercepting network requests can be helpful in various situations: Debugging and troubleshooting network issues. Inspect the request and response payloads. Modify the requests in real-time. By intercepting and modifying network requests, inspecting and manipulating various aspects of the communication, such as headers, parameters, cookies, and response data, is possible. When intercepting network requests, usually have a proxy between the client and the server. The proxy will intercept all network traffic between the two endpoints and allow the user or tool to inspect and modify the traffic before forwarding it to its intended destination. In the diagram above, the client is requesting the server, but the request is intercepted by a proxy before it reaches the server. The proxy can then analyze and modify the request before forwarding it to the server and can also analyze and modify the response before sending it back to the client. One commonly used tool for analyzing network traffic in Wireshark. Wireshark is a free and open-source packet analyzer that lets you capture and analyze network traffic in real-time. Another tool for analyzing network traffic is tcpdump. It can capture and filter network traffic for various protocols, including TCP, UDP, ICMP, and more. Why Intercept Network Requests? Intercepting network requests can provide many benefits, depending on the context and the reason for the interception. Here are a few examples: Debugging: By intercepting network requests, developers can easily see what data is being sent and received during different application stages. This can help to quickly identify and resolve any issues related to the application’s network communication. Testing: Intercepting network requests can be beneficial for automating tests of web applications. Test scripts can be written to simulate user actions and inspect the resulting network traffic, allowing developers to automate the testing of complex workflows and scenarios. Data Modification: Interception can modify network requests, allowing developers to manipulate the data sent over the network. Accelerating test execution: By intercepting network requests and returning mock responses, you can reduce the number of requests your application makes to the server. Security: Intercepting network requests can also be used as a security measure. For example, In organizations, you can monitor network traffic to detect malicious activity, such as a cyber-attack. Modifying network requests: By intercepting network requests, developers can modify the requests made by an application to test different scenarios or to implement new features. By modifying requests and analyzing responses, you can quickly identify and fix issues in your code. Below are some other benefits of intercepting the network request, which is helpful for the network team. Traffic analysis: By capturing and analyzing network requests, network administrators can gain insights into network usage patterns, identify potential performance bottlenecks, and optimize the network for better performance. Content filtering: By intercepting network requests, organizations can filter content based on various criteria, such as security policies, bandwidth limitations, or content type. Monitoring: Interception can also be used for monitoring purposes, for example, to gather statistics about the usage of an application or to gain insights into user behavior. QA Tools for Intercepting Network Requests Here are some QA automation testing tools that can be used for intercepting network requests Cypress: Cypress can intercept network requests and manipulate their responses. In Cypress, you can intercept network requests using the Cypress intercept — cy.intercept() method. This command allows you to intercept and modify network requests and responses. Playwright: With Playwright, you can intercept network requests using the page.route() method. This method allows you to intercept network requests and provide a custom response. Postman: Postman is a popular API development and testing tool that allows you to inspect and modify network requests. It has a user-friendly interface and a wide range of features, making it a great option for QA automation testing. Charles: Charles is a web debugging proxy tool that can inspect, debug, and modify network requests. It is widely used by developers to troubleshoot issues with web applications and APIs. Fiddler: Fiddler is a web debugging proxy tool that allows you to inspect and modify network requests, and it is widely used for QA automation testing. JMeter: JMeter is a popular open-source load testing tool with features for intercepting and modifying network requests. JMeter can be used to test the performance of web applications and APIs. Burp Suite: Burp Suite is a powerful and widely used tool for testing web applications. It allows you to intercept and manipulate HTTP requests and responses and provides detailed information about the requests and responses. SoapUI: SoapUI is an open-source tool for testing web services, including features for intercepting network requests. It is widely used for testing SOAP and REST APIs. Selenium: Selenium is a popular open-source framework for automating web application testing. While Selenium itself does not have built-in capabilities for intercepting network requests. Intercepting Network Requests With Cypress Here are some common use cases for using Cypress intercept for handling network requests. Mocking APIs Cypress allows developers to create mock APIs to simulate different server responses. This is useful when testing application behavior under different conditions, such as when the server is down or the response is delayed. Testing HTTP Requests and Responses By intercepting network requests, the Cypress UI automation tool allows developers to test how the application handles HTTP requests and responses. This includes testing error response handling, response time, and response code. Testing Authorization and Authentication Using Cypress, developers can test how their application handles authentication and authorization by intercepting network requests and passing the authorization tokens. Here are some circumstances under which authorization and authentication are used in intercepting network requests: Access Control: Authorization is used to control access to resources such as files, web pages, APIs, and databases. By requiring authorization, only authorized users or systems are granted access to the resource. Secure Data Transmission: Authentication ensures data is securely transmitted between systems. By authenticating the sender and receiver, data can be encrypted and decrypted only by the intended parties. Testing for Performance-Related Issues Cypress UI testing tool can be used to measure the performance of web applications by intercepting network requests and measuring the response times of each request. This can help developers identify performance bottlenecks in their applications and optimize performance. Using Cypress Intercepts for Handling Network Requests Cypress is a JavaScript-based end-to-end testing framework that makes writing, running, and debugging tests for web applications easy. It has built-in support for intercepting and stubbing network requests, allowing you to control the data returned from the server and make assertions about the network requests made by your application while performing Cypress end-to-end testing. You can use the Cypress intercept — cy.intercept() command to intercept network requests in Cypress. This Cypress intercept command takes a URL pattern and a callback function as arguments and will intercept all requests that match the pattern. In the callback function, you can then modify the request, return a response, or continue the request to the network. Below are the methods you can use in Cypress for Spy and stub network requests and responses. JavaScript cy.intercept(url) cy.intercept(method, url) cy.intercept(routeMatcher) Here’s a simple example of how you could use the Cypress intercept command to return a fake response for a certain request: JavaScript cy.intercept('/api/data', { method: 'GET' }).as('getData') .reply(200, { data: 'Test Data' }); // … Perform your test logic … cy.wait('@getData') .its('response.body') .should('deep.equal', { data: 'Test Data' }); In this example, the Cypress intercept — cy.intercept() command is used to intercept all GET requests to the /api/data endpoint. The .reply method is then used to return a fake response with a status code of 200 and a JSON body of { data: Test Data’ }. The cy.wait() command is then used to wait for the request to be intercepted and complete, and the response body is asserted to match the expected value. Before explaining all the methods in detail, let’s first see how the Cypress intercept — cy.intercept() method works. How Does the cy.intercept() Method Work? The Cypress intercept or cy.intercept() is a method used to intercept and modify HTTP requests and responses made by the application during testing. This allows you to simulate different network scenarios and test the behavior of your application under different conditions. In the diagram below, the Cypress intercept — cy.intercept() method intercepts the requests and responses made by the Application Under Test (AUT). The Cypress intercept method can intercept requests to specific URLs or requests made by specific methods (e.g., GET, POST, etc.) The steps of the diagram are explained below: The client (browser) initiates a request to the server. The server receives the request and sends back a response. The client receives the response and handles it. Cypress test code intercepts the request before it reaches the server. The test code can modify the request or the response in any way it wants, such as adding headers, delaying the response, or changing the status code. The test code returns a stubbed response that replaces the actual response from the server. The client receives the stubbed response and handles it as if it came from the server. Once you get the response, we can verify the stubbed response by explaining the assertion below with various examples. Different Ways of Intercepting Network Requests in Cypress There are various ways to use Cypress intercept for handling network requests: 1. Matching URL The first way of intercepting the request is by Matching the URL. There are three ways of matching the URL. The interception by matching the exact URL. JavaScript it('Intercept by Url', () => { cy.visit('https://reqres.in/'); cy.intercept('https://reqres.in/api/users/').as('posts') cy.get("[data-id=users]").click() cy.wait('@posts').its('response.body.data').should('have.length', 6) }) In this example, we are intercepting the complete reqres.in URL. The intercepted request is assigned a named alias, ‘posts,’ using the .as method. The test then waits for the ‘posts’ request to complete and verifies the length of the response body. 2. Interception of multiple URLs using pattern matching. JavaScript it('Intercept by use pattern-matching to match URLs', () => { cy.visit('https://reqres.in/'); cy.intercept('/api/users/').as('posts') cy.get("[data-id=users]").click() cy.wait('@posts').its('response.body.data').should('have.length', 6) }) In this example, we are intercepting the request having the URL match the /api/users/ pattern. The intercepted request is assigned a named alias, ‘posts,’ using the .as method. The test then waits for the ‘posts’ request to complete and verifies the length of the response body. 3. Interception of the URL using a regex pattern. JavaScript it('Intercept by regular expression', () => { cy.visit('https://reqres.in/'); cy.intercept('/\/api/users?page=2').as('posts') cy.get("[data-id=users]").click() cy.wait('@posts').its('response.body.data').should('have.length', 6) }) In this example, the Cypress intercept — cy.intercept() method intercepts the URL that matches the regex pattern /\/api\/users. The as() method gives a name to the intercepted request, which can be later used with cy.wait() to wait for the response. In this case, cy.wait(‘@posts’) waits for the intercepted request to complete before proceeding with the test. 2. Matching Method Another way of intercepting the request is by Matching the methods. By default, if you don’t pass a method argument, then all HTTP methods (GET, POST, PUT, PATCH, DELETE, etc.) will match. Bypassing the method in the cy.intercept() will intercept a particular method request in a network call. Suppose you have provided an interceptor command like cy.intercept('/api/users/'). In that case, we will match parameters in all Methods (GET, POST, PUT, PATCH, DELETE, etc.) But if you pass the Method name in command cy.intercept ('GET', '/users?page=2'), then, in that case, only the GET method is intercepted. JavaScript it('Intercept by matching GET method', () => { cy.visit('https://reqres.in/'); cy.intercept('GET','api/users?page=2').as('posts') cy.get("[data-id=users]").click() cy.wait('@posts').its('response.body.data').should('have.length', 6) }) Another example of a POST request is where we have manipulated the response by providing the data in the body. In the below example, we have provided the data in the body, and thus, we have mocked the data with provided data in the body. JavaScript it('Intercept by matching POST method', () => { cy.visit('https://reqres.in/'); cy.intercept('POST', 'api/users', (req) => { req.reply({ status: 200, body: { "name": "John", "job": "QA Manager", } }) }).as('updateuser') cy.get("[data-id=post]").click() cy.wait('@updateuser') }) Below is the output of the above test case. You can see we have mocked the data by intercepting the POST call. 3. Matching With RouteMatcher RouteMatcher is a part of the Cypress API that allows you to match specific network requests based on their URL, method, headers, and other attributes. By using a RouteMatcher, you can match requests based on their URL patterns, which provides a flexible way to intercept API requests and test our application’s behavior under different conditions. JavaScript it('Intercept by RouteMatcher ', () => { cy.visit('https://reqres.in/') cy.intercept({ method: 'GET', url: 'https://reqres.in/api/users/**' }, (req) => { req.reply({ statusCode: 200, body: { data: [{ id: 7, email: 'tim.Bluth@reqres.in', first_name: 'tim', last_name: 'Bluth', avatar: 'https://reqres.in/img/faces/1-image.jpg'}, { id: 8, email: 'janet.weaver@reqres.in', first_name: 'Janet', last_name: 'Weaver', avatar: 'https://reqres.in/img/faces/2-image.jpg'}]} }) }).as('postdata') cy.wait('@postdata').its('response.body.data').should('have.length', 2) }) In this example, we have used a RouteMatcher to match any GET request to the https://reqres.in/api/users/** endpoint. The ** notation matches any path after /api/users/. We then use the req.reply() function to return a custom response for matching requests. Finally, we load our application and verify that the response has length 2. Output: The output of the above test case is attached below: 4. Pattern Matching In Pattern matching, you can provide the matching pattern string like in the below example any GET Or PATCH requests that match the pattern **/users/** will be intercepted. JavaScript it('Intercept by Pattern Matching using glob matching ', () => { cy.visit('https://reqres.in/') cy.intercept({ method: '+(GET|PATCH)', url: '**/users/**' }, (req) => { req.reply({ statusCode: 200, body: { data: [{ id: 7, email: 'kim.smith@reqres.in', first_name: 'Kim', last_name: 'Smith', avatar: 'https://reqres.in/img/faces/1-image.jpg'}, { id: 8, email: 'janet.weaver@reqres.in', first_name: 'Janet', last_name: 'Weaver', avatar: 'https://reqres.in/img/faces/2-image.jpg'}]} }) }).as('postdata') cy.wait('@postdata').its('response.body.data').should('have.length', 2) }) Output: In the below code, you can see the data that we have mocked displaying under the response body. 5. Stubbing a Response In Cypress, stubbing a response refers to the process of intercepting a network request made by the application being tested and returning a predefined response instead of the actual response from the server. There are two ways to stub a response for a network request: — With a string Here’s an example of how you can use the Cypress intercept — cy.intercept() method to stub a response for a network request by passing a string in the body. JavaScript it('Stubbing a response With a string', () => { cy.visit('https://reqres.in/') cy.intercept('GET', '**/users/**', { statusCode: 200, body: 'Hello, world!' }).as('getUsers') cy.wait('@getUsers') cy.get('@getUsers').then((interception) => { expect(interception.response.body).to.equal('Hello, world!') }) }) Output: — With Fixture files Another way of stubbing the response using the fixture file. You can mock the data from the fixture file instead of providing data in the body. JavaScript it('Stubbing a response With Fixture file', () => { cy.visit('https://reqres.in/') cy.intercept('GET', 'https://reqres.in/api/users?page=2', { fixture: 'users.json' }).as('getUsers') cy.visit('https://reqres.in/') cy.wait('@getUsers') cy.get('.data').should('have.length', 6) }) In this example, we are intercepting a GET request to the endpoint reqres.in and responding with a fixture file called users.json. We also use the .as() method to assign the intercepted request to an alias to wait for the response before performing further actions. Assuming you have a fixture file named users.json in your cypress/fixtures directory, this test will verify that the .data element on the page has a length of six, which matches the number of records in the users.json fixture file. Output: In the below output of the above code, you can see the data we have mocked using the fixture file. 6. Changing Headers You can also use the Cypress intercept — cy.intercept() method to mock header data. JavaScript it('Intercept a request and modify headers', () => { cy.visit('https://reqres.in'); cy.intercept('GET', 'https://reqres.in/api/users', (req) => { req.headers['Authorization'] = 'Bearer my-token'; }).as('getUserList'); //cy.visit('https://reqres.in/api/users'); cy.wait('@getUserList') cy.get('@getUserList').then((interception) => { const requestHeaders = interception.request.headers; expect(requestHeaders).to.have.property('Authorization', 'Bearer my-token'); }); }); In this example, we intercept a GET request to reqres.in users and modify the Authorization header by adding a token value. We give this interception a unique alias using the .as() command so that we can wait for it to complete using cy.wait(). After waiting for the interception to complete, we use the cy.get(‘@getUserList’) command to get the interception object and assert that the Authorization header was modified correctly. How to Override an Existing Cypress Intercept? Overriding an existing Cypress intercept allows you to modify or cancel an existing network request intercept that has already been defined in your test code. This can be useful when you want to change the behavior of an existing intercept, for example, to simulate a different response from a server or to modify the request data in a different way. The main difference between intercepting network requests and overriding an existing intercept is that intercepting allows you to define new intercepts in your test code while overriding allows you to modify existing intercepts. Cypress provides a rich API for working with both intercepts and overrides to help you thoroughly test and debug your web applications. The below example shows how you can use the Cypress intercept — cy.intercept() method to override an existing intercept and modify the behavior of your application during testing. JavaScript describe.only('Override an existing intercept example', () => { beforeEach(() => { cy.intercept('GET', 'https://reqres.in/api/users').as('getUsers') }) it('overrides the response of the /api/users request', () => { cy.visit('https://reqres.in/') cy.intercept('GET', 'https://reqres.in/api/users', (req) => { req.reply((res) => { res.send({ data: [{ id: 1, email: 'test@test.com' }], page: 1, per_page: 1, total: 1, total_pages: 1 }) }) }).as('getUsers') cy.wait('@getUsers').then((interception) => { expect(interception.response.body.data).to.have.length(1) expect(interception.response.body.data[0].email).to.eq('test@test.com')}) }) }) In this example, we first define an intercept for the GET /api/users request and give it an alias of getUsers. Then, in the test itself, we override the same request by defining a new intercept with the same alias of getUsers. In the new intercept, we use the req.reply() function to override the original request’s response and return a new response that includes a single user with an email of ‘test@test.com‘. Finally, we use the cy.wait() command to wait for the getUsers alias to complete, and then we test the response to ensure it contains the expected data. Output: Wrapping Up Using stubbing, requests to a network are intercepted and replaced with predefined responses, rather than sending the request over the network and waiting for a response. However, stubbing can also lead to false positives, as the behavior of the stubbed requests may not accurately reflect the behavior of the actual requests. This can lead to a false sense of security in your tests and may miss bugs or errors that only occur in the actual network request. On the other hand, not using stubbing with the Cypress intercept — cy.intercept() can provide a more accurate picture of the behavior of your application in the real world. By allowing actual network requests to be made and handling them accordingly, you can be more confident that your tests accurately reflect the behavior of your application in the wild.
All We Can Aim for Is Confidence Releasing features is all about confidence: confidence that features work as expected; confidence that our work is based on quality code; confidence that our code is easily maintainable and extendable; and confidence that our releases will make happy customers. Development teams develop and test their features to the best of their abilities so that quality releases occur within a timeframe. The confidence matrix shown below depicts four main areas: The high confidence and small release time area (an area that all development teams strive for) The low confidence and small release time area The high confidence and long release time area The low confidence and long release time area The first is when we’ve made a quality release quickly. The second is when we quickly released features that may be buggy. The third is when it took us a while to do a quality release. The fourth is when it took us a while to make a buggy release. Think of the confidence matrix as a return on investment (ROI) matrix in its most basic form where our return is confidence and our investment is time. When feature development starts, confidence could be high or low. We may be confident that we know what we must develop and how to do it. I’ve found that most software projects start in the low-confidence zone. New features could mean new unknowns that result in low confidence. Most importantly, as our development and testing activities continue, as our release time reaches the deadline, our confidence should increase. Unfortunately, this is not always the case. To achieve confidence, most teams test and use development best practices. Despite their best efforts, I’ve seen teams releasing fast or slow with high or low confidence. Teams’ confidence may have started low but finished high or vice versa. This article shares experiences about how teams have tried to gain confidence from testing. Confidence From Tests Requires Reliable Tests Tests will either pass or fail. We execute them to get a true picture of the system under test. The system could be a unit or units of code or a complete application. The true picture could be that a new feature is ready to be released or that there are problems that need to be fixed before releasing. Once we’ve got the true picture we can make decisions based on testing results and not guesses. How do we know that we’ve got the true picture? By trusting our testing results. Trusting our testing results means that no matter how many times we execute a test suite, all the tests will have no false positives and no false negatives. Tests should not pass accidentally. For example, if out of ten runs they pass five times and fail five times, they are not reliable. Such testing results are as good as guesses and will not give us a true picture of the system under test. A test may be failing for irrelevant reasons while the functionality that it exercises could be working as expected. We need to have reliable tests where we can trust our test results. No matter how much code we cover with tests, no matter how fast or slow our tests run, we will get confidence from our testing efforts if and only if our tests are reliable. Levels of Testing: Speed vs Scope A simple way to understand scope is the following rule of thumb: large scope means that we cover many lines of code. Small scope means that we cover a few lines of code. Traditionally, there are four testing levels. The lower level is unit testing, followed by integration testing, system testing, and acceptance testing, which is the higher testing level.Unit testing is about making educated decisions about what inputs should be used and what outputs are expected per input. Groups of inputs should be identified that have common characteristics and are expected to be processed in the same way by the unit of code under test. This is known as partitioning, and once such groups are identified, they should be covered by unit tests. Unit tests have a small scope. To cover our code thoroughly we need many unit tests. This is usually not a problem because we can run thousands of them in a few seconds. As we go from lower to higher testing levels the scope increases and test execution speed becomes an issue. Once a unit of code is defined we may also define components of code by grouping code units together. Integration testing is about interactions and interfacing between different components. Compared with unit tests, integration tests have a larger scope, but are roughly at the same order of magnitude when it comes to test execution speed. At a system level, our product is tested at a large scope. A single system test could cover thousands of units and hundreds of components of code. Such tests take time to execute. If we could build confidence without needing thousands of them, then that would be good news. The bad news is that test execution speed is so low that it could prolong our feature releases considerably. Similar to system tests, acceptance testing has a large scope. In some companies, it is performed by customers or company team members at the customer’s site. Other companies use acceptance testing as validation testing performed by the customers. Speed Is Vital To release a feature, we could test to gain confidence that it works as expected, functionally and non-functionally. It takes time to build confidence. We need time to perform development and testing, assess our testing results, and make a decision about releasing or not. Are we good to release or should we fix the bugs we’ve found, redeploy to test that all fixes are OK, and then release? To minimize the feature-release time, we need to minimize at least: The time it takes to develop the feature: Using coding best practices during development is one way to introduce fewer bugs. The time it takes to test: We test to find bugs. Are they important? We should fix and redeploy. Are they not important? We could deploy with known issues. There are teams that fix a bug, deploy the bug fix in a testing environment, test that the bug fix works as expected and that it does not introduce any new issues, and then deploy to production. Others deploy bug fixes directly to production (this is faster but could be riskier). Release speed is vital. Depending on how much time we’ve got to release a feature, I’ve seen teams making various decisions in order to handle deadlines. These included: Features are released without testing while coding standards used for development are questionable. An example of this is a team that usually started and finished their development efforts in the low confidence area. The team has had a hard time understanding why a number of problems have arisen after their releases. Most importantly, the most critical problems remained under their radar for a long time. Features are released without testing while other coding standards are met and developers are confident with their code. There was a team of experienced developers that did not believe in testing. The closer to testing that they would get would be debugging their code. They were usually between the high confidence/small release time and high confidence/long release time areas in the confidence matrix. Bugs could have fallen under their radar occasionally and testers from other teams would be brought for QA testing when the team was about to release features with rich functionality. Features are released with just a few unit or integration-level tests but a large number of UI tests. This is a case that I’ve seen many times. Such teams would fall in any of the four areas in the confidence matrix. When showstopper bugs were found late by the testers and when fixing them required major rewrites from developers, the team's confidence was low and the release deadlines may have been prolonged. Even if no showstoppers were found, testing was a bottleneck. Developers were reluctant to change the code in a number of areas and each change called for extensive regression testing at the UI from QA testers. When releasing features with rich functionality, QA testing at the UI was a bottleneck because the test execution speed was low and the tests were many. UI test automation has helped to overcome this problem for some teams while for other teams it gave a smaller ROI than expected. Features are released with a large number of unit and integration tests and a minimal set of UI tests. Such teams would usually fall in the high confidence/small release time area of the confidence matrix. Bugs may occasionally have gone under the radar, especially for features with rich functionality but they were usually fixed quickly without side effects. They had continuous integration and continuous deployment setup. Their continuous builds were made of unit tests and integration tests. Frequently executing unit and integration tests was the main source of their confidence. A final confidence boost was given by a small number of manual exploratory tests in the UI. Features released with a large number of integration tests, a number of unit tests, and a few UI tests. This was the case for teams that used microservices and teams that executed a large number of front-end tests. Some JavaScript frontend developers, for example, were strong believers of the “write tests, not too many, mostly integration” paradigm. In the case of backend developers writing microservices, they believed that in a world of microservices, the biggest complexity is not within the microservice itself, but in how it interacts with others. As a result, they gave special attention to writing tests exercising interactions between microservices. Such teams usually avoided the low confidence and long release time area in the confidence matrix. Following good coding standards and best coding practices does not mean that we should not test. In fact, testing is another best practice for coding. As this article focuses on testing and not other coding standards, it suffices to mention that testing is always a good idea. However, when developing and testing, our release speed will be affected by our testing speed, too. Testing dynamics per testing level need to be taken into account, in order to get the most value from our testing efforts for the allocated time. Test Execution Speed Testing at any level is important and necessary. The lower the testing level the faster the test execution speed. I’ve witnessed at least three ways that test execution speed has affected how development teams work. To identify what compromises to make: If we must make compromises, make an educated decision about what to do and what to avoid. Depending on how much time we have for testing, I've seen teams choosing at what testing level they should test. Ideally, if time and costs were not a constraint, we should test at all levels possible. This is because 100% test coverage at a unit level does not mean that we will catch no bugs with integration testing and/or with system testing. The same is true for each testing level. However, a test suite of 1000 unit tests may take an hour to complete while a UI automation suite with 200 tests may take one day to complete. Although choosing not to test at any level may involve risks, if we have little time to dedicate to testing we may make educated decisions according to what tests we want to run and at what level. To identify how fast we will get feedback from our tests: The test result is our feedback. Did the test pass? Our feedback is a green light. Did the test fail? Our feedback is a red light. A development team tested first the most important functional and non-functional areas of their release. They first tested at a testing level that test execution speed was the fastest. As a result, showstopper bugs could be found early during testing and hence they were also fixed early without jeopardizing release time. The main factor that lowered their confidence was showstoppers that were found late and fixed late, resulting in missing release deadlines. They’ve found that the best way to allocate their testing efforts was to start with quick feedback testing (unit and integration testing). If no showstoppers are found, then for the remaining time, continue with higher-level testing. To help identify our testing levels: People often go back and forth about whether particular tests are unit tests or integration tests. Large unit tests could also be considered small integration tests and vice-versa. But what are they really and at what level do they belong? There was a team that shared a definition like, "If a test talks to the database or if it communicates across the network, if it involves accessing file systems like editing configuration files, then it’s not a unit test." The reasoning behind this was simple: test execution speed. If a test talked to the database, for example, then it would take longer to execute. Since unit tests are the fastest across all testing levels, the team decided to call low-level tests that performed such time-consuming actions as integration tests. Another team was using fault detection time as a guide. Α test failed. If it took seconds to detect the fault in the code that caused the failure, then the failing test was a unit test. If it took minutes to detect the fault then the failing test was an integration test. There was a group that used architects and tech leads to write a few integration tests. Their main goal was to ensure that the choreography and orchestration of the architectural components were working. Such tests usually covered 10 to 20% of the code at maximum and having a large scope they usually were slow. In another group, QA and business analysts wrote acceptance tests to achieve a maximum of 50% of code coverage. They also wrote a few system tests as final tests of choreography and orchestration. The system tests covered very little of the actual business rules and were the slowest. Wrapping Up There is a popular debate about what percentage of tests to write at what testing level. I’ve tried to shift the focus a little bit on confidence over time. It’s all about confidence, and a great deal of it can be achieved by running tests quickly and reliably. Testing closer to the unit/integration level will be quicker and necessary, but not sufficient. Higher testing levels will also need to be covered which will probably cost more in execution time, maintenance, and reliability. Let’s not forget one of our basic prerequisites for tests to be valuable: tests that pass for the right reasons and fail for useful reasons. I’ve shared a number of experiences about how different development teams managed their testing efforts resulting in different levels of confidence over time.
Microservices architecture is an increasingly popular approach to building complex, distributed systems. In this architecture, a large application is divided into smaller, independent services that communicate with each other over the network. Microservices testing is a crucial step in ensuring that these services work seamlessly together. This article will discuss the importance of microservices testing, its challenges, and best practices. Importance of Microservices Testing Testing microservices is critical to ensuring that the system works as intended. Unlike traditional monolithic applications, microservices are composed of small, independent services that communicate with each other over a network. As a result, microservices testing is more complex and challenging than testing traditional applications. Nevertheless, testing is crucial to detect issues and bugs in the system, improve performance, and ensure that the microservices work correctly and efficiently. Microservices testing is critical for ensuring a microservices-based application's reliability, scalability, and maintainability. Here are some reasons why microservices testing is essential: Independent Testing: Each microservice is an independent unit, which means that it can be tested separately. This makes testing easier and more efficient. Increased Agility: Testing each microservice separately allows for faster feedback and faster development cycles, leading to increased agility. Scalability: Microservices can be scaled horizontally, which means that you can add more instances of a service to handle increased traffic. However, this requires proper testing to ensure that the added instances are working correctly. Continuous Integration and Delivery: Microservices testing can be integrated into continuous integration and delivery pipelines, allowing for automatic testing and deployment. Challenges of Microservices Testing Testing microservices can be challenging due to the following reasons: Integration Testing: Testing the interaction between multiple microservices can be challenging because of the large number of possible interactions. Network Issues: Microservices communicate with each other over the network, which can introduce issues related to latency, network failure, and data loss. Data Management: In a microservices architecture, data is often distributed across multiple services, making it difficult to manage and test. Dependency Management: Microservices can have many dependencies, which can make testing complex and time-consuming. Best Practices for Microservices Testing Here are some best practices for microservices testing: Test Each Microservice Separately: Each microservice should be tested separately to ensure that it works as expected. Since microservices are independent services, it is essential to test each service independently. This allows you to identify issues specific to each service and ensure that each service meets its requirements. Use Mocks and Stubs: Use mocks and stubs to simulate the behavior of other services that a service depends on. Mock services are useful for testing microservices that depend on other services that are not available for testing. Mock services mimic the behavior of the missing services and allow you to test the microservices in isolation. Automate Testing: Automate testing as much as possible to speed up the process and reduce human error. Automated testing is essential in a microservices architecture. It allows you to test your system repeatedly, quickly, and efficiently. Automated testing ensures that each service works independently and that the system functions correctly as a whole. Automated testing also helps to reduce the time and effort required for testing. Use Chaos Engineering: Use chaos engineering to test the resilience of your system in the face of unexpected failures. Test Data Management: Test data management and ensure that data is consistent across all services. Use Containerization: Use containerization, such as Docker, to create an isolated environment for testing microservices. Test Service Integration: While testing each service independently is crucial, it is equally important to test service integration. This ensures that each service can communicate with other services and that the system works as a whole. In addition, integration testing is critical to detecting issues related to communication and data transfer. Test for Failure: Failure is inevitable, and microservices are no exception. Testing for failure is critical to ensure that the system can handle unexpected failures, such as server crashes, network failures, or database errors. Testing for failure helps to improve the resilience and robustness of the system. Conclusion Microservices testing is a critical step in ensuring the reliability, scalability, and maintainability of microservices-based applications. Proper testing helps to identify issues early in the development cycle, reducing the risk of costly failures in production. Testing each microservice separately, automating testing, testing each service independently, testing service integration, testing for failure, and using mocks and stubs are some best practices for microservice testing. By following these best practices, you can ensure that your microservices-based application is reliable and scalable. In addition, implementing these best practices can help to improve the reliability, resilience, and robustness of your microservices architecture.
The relevance of using a BDD framework such as Cucumber.js is often questioned by our fellow automation testers. Many feel that it is simply adding more work to their table. However, using a BDD framework has its own advantages, ones that can help you take your Selenium test automation a long way. Not to sideline, these BDD frameworks help all of your stakeholders to easily interpret the logic behind your test automation script. Leveraging Cucumber.js for your Selenium JavaScript testing can help you specify an acceptance criterion that would be easy for any non-programmer to understand. It could also help you quickly evaluate the logic implied in your Selenium test automation suite without going through huge chunks of code. With a given-when-then structure, Behaviour Driven Development frameworks like Cucumber.js have made tests much simpler to understand. To put this into context, let’s take a small scenario. You have to test an ATM if it is functioning well. We’ll write the conditions that given the account balance is $1,000 and the card is valid, and the machine contains enough money. When the account holder requests $200, the cashpoint should dispense $200 and the account balance should be $800, and the card should be returned. In this Cucumber.js tutorial, we’ll take a deep dive into a clear understanding of the setup, installation, and execution of our first automation test with Cucumber.js for Selenium JavaScript testing. What Is Cucumber.js and What Makes It So Popular? Let’s start our Cucumber.js tutorial with a small brief about the framework. Cucumber.js is a very robust and effective Selenium JavaScript testing framework that works on the Behavior Driver Development process. This test library provides easy integration with Selenium and provides us the capability to define our tests in simple language that is even understood by a layman. Cucumber.js Selenium JavaScript testing library follows a given-when-then structure that helps in representing the tests in plain language, which also makes our tests to be a point of communication and collaboration. This feature improves the readability of the test and hence helps understand each use case in a better way. It is used for unit testing by the developers but majorly used for integration and end-to-end tests. Moreover, it collaborates with the tests and makes them so legible that there is hardly a need for documentation for the test cases, and can even be digested by business users. Setting up Cucumber.js for Selenium Javascript Testing So, before we carry on with our Cucumber.js Tutorial, to begin writing and executing our automated test scripts using Cucumber, we need to set up our system with the Cucumber.js framework and install all the necessary libraries and packages to begin Selenium JavaScript testing. Node JS and Node Package Manager (npm): This is the foundation package and most important for any Selenium Javascript testing framework. It can be downloaded via the npm manager, i.e., by installing the node package manager from the nodejs.org official website or using the package installer for different operating systems downloaded from the website here for Mac OS, Windows, or Linux. We can execute the npm command in the command line and check if it is correctly installed on the system. Cucumber.js Library Module: The next prerequisite required for our test execution is the Cucumber.js library. We would need the Cucumber.js package as a development dependency. After the successful installation and validation of the Node.js on the system, we would use the node package manager, i.e., npm, that it provides to install the Cucumber.js library package in the npm repository. So, in order to install the latest version of the Cucumber.js module, we will use the npm command as shown below $ npm install -g cucumber and npm install --save-dev cucumber Here the parameter g indicates the global installation of the module, which means that it does not limit the use of the module to the current project, and it also can be accessed with command-line tools. The command executed using the parameter –save-dev will place the Cucumber executable in the base directory, i.e., ./node_modules/.bin directory, and execute the commands in our command-line tool by using the cucumber keyword. Java – SDK: Since all the Selenium test framework internally uses Java, next, we would move ahead and install the Java Development Kit on our systems. It is advised to use JDK having version 6.0 and above and set up/configure the system environment variables for Java. Selenium Web Driver: To automate the system browser, we would need to install Selenium Web Driver Library using the below npm command. In most cases, it automatically gets installed in our npm node_modules base directory as a dependency when installing other libraries. $ npm install selenium-webdriver Browser Driver: Finally, it is required to install the browser driver. It can be any browser where we would want to execute the test scenarios, and hence the corresponding driver needs to be installed. This executable needs to be added to our PATH environment variable and also placed inside the same bin folder. Here we are installing the chrome driver. Here is the link to the documentation where we can find and download the version that matches the version of our browser. $ npm install -g chromedriver Working on the Cucumber.js Test Framework? Now that we’ve set up our system for our Cucumber.js tutorial, we will move ahead with creating our project structure and create a directory named cucumber_test. Then we will create two subfolders, i.e., feature and step_definition, which will contain respective scripts written for our features and step definition. $ mkdir step_definitions Finally, the folder will have a generated package.json file in the base directory of the package and save all of the dev dependencies for these modules. Another important thing to do with the package.json file is to add the test attribute in the scripts parameter. { "scripts": { "test": "./node_modules/.bin/cucumber-js" } } By adding this snippet to our package.json file, we can run all your cucumber tests from the command line by just typing in “npm test” on the command line. Our final project folder structure stands as below. cucumber_test | - - feature | - - feature_test.feature | - - step_definition | - - steps_def.js | - - support | - - support.js | - - package.json Below is the working procedure of a Cucumber.js project: We begin with writing a .feature file that has the scenarios and each of these scenarios with a given-when-then structure defined. Next, we write down step definition files which typically define the functions that match the steps in our scenarios. Further, we implement these functions as per our requirement or automate the tests in the browser with the Selenium driver. Finally, we run the tests by executing the Cucumber.js executable file present in the node_modules/.bin folder. Running Our First Cucumber.js Test Script The next step in this Cucumber.js tutorial is to execute a sample application. We will start by creating a project directory named cucumber_test and then a subfolder name script with a test script name single_test.js inside it. Then we’ll add a scenario with the help of a .feature file. Which will be served to our application as we instruct Cucumber.js to run the feature file. Finally, the Cucumber.js framework will parse the file and call the code that matches the information in the feature file. Here for our first test scenario, we will start with a very easy browser-based application that, on visiting the Selenium official homepage, allows us to make a search by clicking the search button. Please make a note of the package.json file that we will be using in our upcoming demonstrations. package.json The package.json file contains all the configurations related to the project and certain dependencies that are essential for the project setup. It is important to note that the definitions from this file are used for executing the script, and hence this acts as our project descriptor. JSON { "name": "Cucumber.js Javascript test with Selenium", "version": "1.0.0", "description": "CucumberJS Tutorial for Selenium JavaScript Testing", "main": "index.js", "scripts": { "test": "./node_modules/.bin/cucumber-js" }, "repository": { "type": "git", "url": "" }, "author": "", "license": "ISC", "description": { "url": "" }, "homepage": "", "dependencies": { "assert": "^1.4.1", "chromedriver": "^2.24.1", "cucumber": "^1.3.0", "geckodriver": "^1.1.3" }, "devDependencies": { "selenium-webdriver": "^3.6.0" } } Now, the first step of the project is to define the feature that we are going to implement, i.e., within this file; we will describe the behavior that we would want from our application, which is visiting the website in our case. This feature lets the browser check for the elements. Hence, we will update our feature file with the code. Below is what our feature file looks like, which contains the given when and then scenarios. feature_test.feature Now we will have our first basic scenarios for visiting the website defined in the feature file followed by other scenarios. These scenarios will follow a given-when-then template. Given: It sets the initial context or preconditions. When: This resembles the event that is supposed to occur in the scenario. Then: This is the expected outcome of the test scenario. steps_def.js Now moving on to define the steps. Here we define the functions that match the steps in our scenarios and the actions that they should perform whenever a scenario is triggered. JavaScript /* This Cucumber.js tutorial file contains the step definition or the description of each of the behavior that is expected from the application */ 'use strict'; const { Given, When, Then } = require('cucumber'); const assert = require('assert') const webdriver = require('selenium-webdriver'); // // The step definitions are defined for each of the scenarios // // // // The “given” condition for our test scenario // // Given(/^I have visited the Selenium official web page on "([^"]*)"$/, function (url, next) { this.driver.get('https://www.selenium.dev').then(next); }); // // The “when” condition for our test scenario // // When(/^There is a title on the page as "SeleniumHQ Browser Automation" "([^"]*)"$/, function (titleMatch, next) { this.driver.getTitle() .then(function(title) { assert.equal(title, titleMatch, next, 'Expected title to be ' + titleMatch); // // The “then” condition for our test scenario // // Then(/^I should be able to click Search in the sidebar $/, function (text, next) { this.driver.findElement({ id: 'searchText' }).click(); this.driver.findElement({ id: 'searchText' }).sendKeys(text).then(next); }); An important thing to note here is that if we execute the test with only written the .feature file and nothing else, the cucumber framework will throw us an error and prompt us to define the steps. It states that although we have defined the feature, the step definitions are missing, and it will further suggest we write the code snippets that turn the phrases defined above into concrete actions. support.js The support and hooks file is used with the step definition to initialize the variables and perform certain validations. JavaScript // // This Cucumber.js tutorial support file to perform validations and initialization for our app // // const { setWorldConstructor } = require('cucumber') const { seleniumWebdriver } = require('selenium-webdriver'); var firefox = require('selenium-webdriver/firefox'); var chrome = require('selenium-webdriver/chrome'); class CustomWorld { constructor() { this.variable = 0 } function CustomWorld() { this.driver = new seleniumWebdriver.Builder() .forBrowser('chrome') .build(); } setWorldConstructor(CustomWorld) module.exports = function() { this.World = CustomWorld; this.setDefaultTimeout(30 * 1000); }; hooks.js It releases the driver when the test execution is complete. JavaScript module.exports = function() { this.After(function() { return this.driver.quit(); }); }; Finally, when we execute the test, we can see in the command line that our test got executed successfully.$ npm test Now let’s have a look at another example that will perform a search query on google and verify the title of the website to assert whether the correct website is launched in the browser. feature_test2.feature Feature: A feature to check on visiting the Google Search website Scenario: Visiting the homepage of Google.com Given I have visited the Google homepage. Then I should be able to see Google in the title bar. steps_def2.js JavaScript /* This Cucumber.js Tutorial file contains the step definition or the description of each of the behavior that is expected from the application which in our case is the webpage that we are visiting for selenium javascript testing .*/ var assert = require('assert'); // // This scenario has only “given” and “then” condition defined // // module.exports = function () { this.Given(/^I have visited the Google homepage$/, function() { return this.driver.get('http://www.google.com'); }); this.Then(/^I should be able to see Google in title bar$/, function() { this.driver.getTitle().then(function (title) { assert.equal(title, "Google"); return title; }); }); }; support2.js JavaScript // This Cucumber.js tutorial support file is used to perform validations and initialization for our application // var seleniumWebdriver = require('selenium-webdriver'); var firefox = require('selenium-webdriver/firefox'); var chrome = require('selenium-webdriver/chrome'); function CustomWorld() { this.driver = new seleniumWebdriver.Builder() .forBrowser('chrome') .build(); } module.exports = function() { this.World = CustomWorld; this.setDefaultTimeout(30 * 1000); }; hooks2.js JavaScript module.exports = function() { this.After(function() { return this.driver.quit(); }); }; Again, when we execute the test, we can see in the command line that our test got executed successfully.$ npm test Kudos! You have successfully executed your first Cucumber.js script for Selenium test automation. However, this Cucumber.js tutorial doesn’t end there! Now that you are familiar with Selenium and Cucumber.js, I want you to think about the scalability issues here. So far, you have successfully executed the Cucumber.js script over your operating system. However, if you are to perform automated browser testing, how would you go about testing your web application over hundreds of different browsers + OS combinations? You can go ahead and build a Selenium Grid to leverage parallel testing. However, as your test requirements will grow, you will need to expand your Selenium Grid, which would mean spending a considerable amount of money over the hardware. Also, every month a new browser or device will be launched in the market. To test your website over them, you would have to build your own device lab. All of this could cost you money and time in maintaining an in-house Selenium infrastructure. So what can you do? You can leverage a Selenium Grid on-cloud. There are various advantages to choosing a cloud-based Selenium Grid over a local setup. The most pivotal advantage is that it frees you from the hassle of maintaining your in-house Selenium infrastructure. It’ll save you the effort to install and manage unnecessary virtual machines and browsers. That way, all you need to focus on is running your Selenium test automation scripts. Let us try and execute our Cucumber.js script over an online Selenium Grid on-cloud. Running Cucumber.js Script Over an Online Selenium Grid It is time that we get to experience a cloud Selenium Grid by getting ourselves trained on executing the test script on LambdaTest, a cross-browser testing cloud. LambdaTest allows you to test your website on 3000+ combinations of browsers & operating systems hosted on the cloud. Not only enhancing your test coverage but also saving your time around the overall test execution. To run the same script on LambdaTest Selenium Grid, you only need to tweak your Selenium JavaScript testing script a little. As you would now want to specify the hub URL for the Remote WebDriver, which would execute your script on our Selenium Grid. Add the username and access key token. For this, we must add the access key token and also the username details in the configuration files, i.e., cred.conf.js file present within the conf directory. The username and access key token can be exported in two ways, as mentioned below. cred.conf.js JavaScript exports.cred = { username: process.env.LT_USERNAME || 'rahulr', access_key: process.env.LT_ACCESS_KEY || 'AbcdefgSTAYSAFEhijklmnop' } Alternatively, the username and access key token can be easily exported using the command as shown below. JavaScript export LT_USERNAME=irohitgoyal export LT_ACCESS_KEY= AbcdefgSTAYSAFEhijklmnop Next, we will look at the feature file. We will be executing our test on the Google Chrome browser. In our test case, we will open the LambdaTest website to perform certain operations on it, such as launching the search engine, validating the content, etc. So, our directory structure will be pretty simple as below: feature_test.feature Now, we need to think of our desired capabilities. We can leverage LambdaTest Selenium Desired Capabilities Generator feature to select the environment specification details which allows us to pick from various combinations that it offers; we can use this to select the combination we want to perform our Selenium javascript testing for this Cucumber.js tutorial. So, in our test scenario the desired capabilities class will look something similar as below: JavaScript const desiredCapabilities = { 'build': 'Cucumber-JS-Selenium-Webdriver-Test', // the build name that is to be display in the test logs 'browserName': 'chrome', // the browser that we would use to perform test 'version':'74.0', // the browser version that we would use. 'platform': 'WIN10', // The type of the Operating System that we would use 'video': true, // flag to check whether to capture the video selenium javascript testing . 'network': true, // flag to check whether to capture the network logs 'console': true, // flag to check whether to capture the console logs 'visual': true // flag to check whether to the capture visual for selenium javascript testing }; With that set, we now look at the step definitions and the cucumber runner.js. step_def.js JavaScript /* This Cucumber.js tutorial file contains the step definition or the description of each of the behavior that is expected from the application which in our case is the webpage that we are visiting. It is aligned with the feature file and reads all the instructions from it and finds the matching case to execute it for selenium javascript testing . */ 'use strict'; const assert = require('cucumber-assert'); const webdriver = require('selenium-webdriver'); module.exports = function() { this.When(/^I visit website of Google on "([^"]*)"$/, function (url, next) { this.driver.get('https://google.com ').then(next); }); this.When(/^the homepage has the field with "Google Search" is present $/, function (next) { this.driver.findElement({ name: 'li1' }) .click().then(next); }); this.When(/^the homepage has the field with "I’m Feeling Lucky" is present $/, function (next) { this.driver.findElement({ name: 'li3' }) .click().then(next); }); this.When(/^I move the cursor and select the textbox to make a search on Google $/, function (text, next) { this.driver.findElement({ id: 'buttonText' }).click(); this.driver.findElement({ id: 'buttonText' }).sendKeys(text).then(next); }); this.Then(/^click the "Google Search" on the text box "([^"]*)"$/, function (button, next) { this.driver.findElement({ id: button }).click().then(next); }); this.Then(/^I must see title "Google" on the homepage "([^"]*)"$/, function (titleMatch, next) { this.driver.getTitle() .then(function(title) { assert.equal(title, titleMatch, next, 'Expected title to be ' + titleMatch); }); }); }; cucumber-runner.js JavaScript #!/usr/bin/env/node // It resembles our runner file for parallel tests. This file is responsible to create multiple child processes, and it is equal to the total number of test environments passed for selenium javascript testing . // let childProcess = require ('child_process') ; let configFile = '../conf/' + ( process.env.CONFIG_FILE || 'single' ) + '.conf.js'; let config = require (configFile ).config; process.argv[0] = 'node'; process.argv[1] = './node_modules/.bin/cucumber-js'; const getValidJson = function(jkInput) { let json = jkInput; json = json.replace(/\\n/g, ""); json = json.replace('\\/g', ''); return json; }; let lt_browsers = null; if(process.env.LT_BROWSERS) { let input = getValidJson(process.env.LT_BROWSERS); lt_browsers = JSON.parse(input); } for( let i in (lt_browsers || config.capabilities) ){ let env = Object.create( process.env ); env.TASK_ID = i.toString(); let p = childProcess.spawn('/usr/bin/env', process.argv, { env: env } ); p.stdout.pipe(process.stdout); } Now since our test scripts are ready to be executed in the cloud grid, the final thing that we are required to do is to run the tests from the base project directory using the below command: $ npm test This command will validate the test cases and execute our test suite across all the test groups that we have defined. And if we open the LambdaTest Selenium Grid and navigate to the automation dashboard, we can check that the user interface shows that the test ran successfully and passed with positive results. Below is the sample screenshot: Don’t Forget to Leverage Parallel Testing Parallel testing with Selenium can help you significantly trim down your test cycles. Imagine if we have at least 50 test cases to execute, and each of them runs for an average run time of 1 minute. Ideally, it would take around 50 minutes to execute the test suite. But if we execute 2 test cases in 2 parallel concurrent sessions, the total test time drops down to 25 minutes. Hence, we can see a drastic decrease in test time. To execute the parallel testing with Selenium for this Cucumber.js tutorial, execute the below command $ npm run parallel. Bottomline Cucumber.js provides us the capability to write tests in a way that is easily read by everyone. Making the framework very flexible and enabling us to create human-readable descriptions of user requirements as the basis for web application tests. With Cucumber.js, we can interact with our webpage on the browser and make various assertions to verify that the changes we performed are actually reflected in our web application on every Browser-OS combination by utilizing Selenium Grid. Still, there is a lot more that can be done with Cucumber.js. Since this test framework is developed over the Selenium interface, it empowers us with limitless capabilities in terms of Selenium JavaScript testing. Let us know if you liked this Cucumber.js tutorial and if there’s any topic you want us to write on. Happy Testing, and Stay Safe!
Justin Albano
Software Engineer,
IBM
Thomas Hansen
CTO,
AINIRO.IO
Soumyajit Basu
Senior Software QA Engineer,
Encora
Vitaly Prus
Head of software testing department,
a1qa