In this Blog will learn How to create a BDD automation framework using Cucumber in JavaScript and Playwright. The playwright is an open-source automation tool. Playwright offers several features that make it stand out are Multi-browser support, Automatic waiting Headless and headful mode, etc. Cucumber is a popular open-source BDD(Behavior Driven Development) testing framework, Which helps build a framework that can be easily understood by both technical and non-technical stakeholders.
Prerequisite:
To get started with Cucumber BDD with playwright-js we need to have the following things installed.
2)Node.js: It is a cross-platform JavaScript runtime environment that allows developers to run JavaScript code outside of a web browser. To get Node.js you can visit: https://nodejs.org/en/download To check Node.js is installed on your system you can use the command ‘node –version’ on CMD or VS Code Terminal.
3)Cucumber (Gherkin) Full Support Extention: This extension provides support for the Cucumber (Gherkin) language to VS Code. To get the extension click on the extension icon on the left side panel of VS Code, search for the extension and install it.
Project setup:
Before starting with framework development we need to create a folder structure.
Create a folder structure with the following steps: 1) Create a folder as ‘Playwright-JS-Demo’, 2) In VS Code open the Playwright-JS-Demo folder 3) Install Playwright and Cucumber by executing the following commands in terminal a. npm init playwright@latest
b.npm i @cucumber/cucumber After execution of these commands, you can see package.json, node_modules, and playwright.config.js is created in the folder structure.
4) once you open the project, create a folder structure as below.
As our project structure is ready, we can start with the framework.
1. TestHooks.js:
Test Hooks mainly contain methods to execute Before and After every execution.
The Before hook gets executed before each Scenario in a cucumber test suite.
In this hook, a new Chrome browser instance is launched for every Test Case.
A new page is created in context with ‘await context.newPage()‘ and assigned to the global variable ‘page‘, which is accessible for any Test Case. This ensures that the browser page is available for each scenario in the test suite.
Once execution is done After the method gets executed It closes the current browser instance.
With Cucumber-BDD we can write scenarios in understandable language. Here I have created a Scenario for OrangeHRM Login.
We can create a separate scenario for each functionality.
Every step in a Feature File describes the action we are going to perform on UI.
In the feature file, we can add specific tags for scenarios or complete feature file ex. @login, @smoke.
If you add a tag for a specific scenario then it will only execute the particular scenario. But if you add tags for the Feature It will execute all the Scenarios from the Feature File.
It will make test execution easy if you want to execute test cases for specific functionality.
Feature: Login Feature
@login
Scenario: Login to OrangeHRM
When I Visit the OrangeHRM login page
And I enter username
And I enter Password
And I click on Login button
Then I verify dashboard URL
3. LoginSteps.js:
Essentially, the purpose of the step file is to attach steps from the feature file to the page file, where actual implementation is available.
We use the “Given-When-Then” (BDD) format to write step definitions. The required statement imports the necessary modules like:
Cucumber library that contains definitions for When Then etc.
A custom LoginPage module that likely contains functions for interaction with the login page.
The steps with ‘When’ are related to the user actions like navigation, clicking on the button, and filling in the information in input boxes, etc. The steps with ‘Then’ are related to the verifications or Assertions, just like in this case I have verified if the Login is successful.
The page file contains the actual implementation of the scenario, We also defined all the functions needed for test execution. Additionally, the LoginPage class contains functions that interact with the login page elements, like navigation to the Login page, entering the username and password, clicking the login button, and verifying the dashboard URL. Moreover, the playwright provides different functions to handle the UI elements and pages. For this test case I have used goto(), click(), fill() waitFor().
process.env.WEB_URL, process.env.WEB_USERNAME and process.env.WEB_PASSWORD are the variables we are accessing from the .env file. (The use of the .env file is explained below in point no 6).
To access the .env file I have imported const path = require(‘path’);
In this file, we specify the paths of all the required files. It helps to identify files in the framework. Whenever you create any new file in the framework, you have to add a path for that new file in this Cucumber.json file so that it can be accessible during the test case execution.
.env file is a Configuration file that contains environment variables. Using a .env file is a best practice for keeping sensitive information separate from the code. Moreover, for this framework, we have stored information like URL to navigate and username and password to Login into the OrangeHRM Web application.
We are executing the ‘npx’ command to run the cucumber-js package which is a test framework for BDD. 1) –require ./steps/*.js specifies The step files from the specified path that should be loaded. 2)–tags @login specifies scenarios with @login tags are going to be executed. You can add more than one tag if needed.
3)–publish flag specifies that test results should be published to the Cucumber Cloud which is a service for storing and analyzing test results.
To start execution you can execute the command ‘npm run test’ in the terminal. Once execution is completed you can see the link for cucumber reports is available in the terminal. By clicking on this link you can see the execution report.
This is how the report looks when you click on the above URL
In conclusion, the BDD automation framework using Cucumber in JavaScript and Playwright helps improve the quality and efficiency of their testing process. This framework will help you to write test cases for any web application very efficiently, It also provides a great reusability of code.
Priyanka is an SDET with 2.5+ years of hands-on experience in Manual, Automation, and API testing. The technologies she has worked on include Selenium, Playwright, Cucumber, Appium, Postman, SQL, GitHub, and Java. Also, she is interested in Blog writing and learning new technologies.
Automating API test suite execution through CI/CD pipelines provides a significant advantage over local execution. By leveraging CI/CD, teams can obtain test results for all systems, improving the speed, quality, and reliability of tests. Manual triggering of API suite execution is not required, freeing up valuable time for team members.
In this blog post, we will guide you through the creation of a workflow file using GitHub Actions for your automated API tests. However, before diving into the creation of a CI/CD workflow, it’s essential to understand some crucial points for a better grasp of the concept.
Before we start creating a CI/CD workflow for our API tests I will suggest you first go through the API test automation framework here and also read this blog on creating a web test automation framework as it helps you to understand the different points which we all should consider before selecting the test automation framework. The API test automation framework is in Python language and has Behave library for BDD purposes.
Let’s understand some basic and important points to start with the CI/CD workflow.
What is DevOps?
DevOps is a set of practices and tools that integrate and automate tasks in the software development and IT industry. It establishes communication and collaboration between development and operations teams, enabling faster and more reliable software build, testing, and release processes. DevOps is a methodology that derives its name from the combination of “Development” and “Operations.”
The primary goal of DevOps is to bridge the gap between development and operations teams by fostering a culture of shared responsibility and collaboration. This helps to reduce the time it takes to develop, test, and deploy software while maintaining high quality and reliability standards. By automating manual processes and eliminating silos between teams, DevOps enables organizations to respond more quickly to changing market demands and customer needs.
CI/CD refers to Continuous Integration and Continuous Delivery, which are processes and practices that help to deliver code changes more frequently and reliably. These processes involve automating the building, testing, and deployment of code changes, resulting in faster and higher-quality software releases for end-users.
The CI/CD pipeline follows a workflow that starts with continuous integration (CI), followed by continuous delivery (CD). The CI process involves integrating code changes into a shared repository and automatically building and testing them to identify errors early in the development process. Once the code has been tested and approved, the CD process takes over and automates the delivery of code changes to production environments.
The CI/CD pipeline workflow helps to reduce the risks and delays associated with manual code integration and deployment while ensuring that the changes are tested and delivered quickly and reliably. This approach enables organizations to innovate faster, respond more quickly to market demands, and improve overall software quality.
Process:
What are GitHub Actions?
GitHub Actions is a feature that makes it easy to automate software workflows, including world-class CI/CD capabilities. With GitHub Actions, you can build, test, and deploy your code directly from GitHub, while also customizing code reviews, branch management, and issue-triaging workflows to suit your needs.
The GitHub platform offers integration with GitHub Actions, providing flexibility for customizing workflows to automate tasks such as building, testing, and deploying code. Developers can create custom workflows using GitHub Actions that are automatically triggered when specific events occur, such as code push, pull request merge, or as per a defined schedule.
Workflows are defined using YAML syntax, which is a human-readable data serialization language. YAML is commonly used for configuration files and in applications to store or transmit data. To learn more about YAML syntax and its history, please visit the following link.
Advantages / Benefits of using GitHub Actions for CI/CD Pipeline:
Seamless integration: GitHub Actions seamlessly integrates with GitHub repositories, making it easy to automate workflows and tasks directly from the repository.
Highly customizable: GitHub Actions offers a high degree of customization, allowing developers to create workflows that suit their specific needs.
Time-saving: GitHub Actions automates many tasks in the software development process, saving developers time and reducing the potential for errors.
Flexible: GitHub Actions can be used for a wide range of tasks, including building, testing, and deploying applications.
Workflow visualization: GitHub Actions provides a graphical representation of workflows, making it easy for developers to visualize and understand the process.
Large community: GitHub Actions has a large and active community, providing a wealth of resources, documentation, and support for developers.
Cost Saving: GitHub Actions come bundled with Github free and enterprise licenses reducing the cost of maintaining separate CI/CD tools like Jenkins
Framework Overview:
This is a BDD API automation testing framework. The reason behind choosing the BDD framework is simple it provides you the following benefits over other testing frameworks.
Improved Collaboration
Increased Test coverage
Better Test Readability
Easy Test Maintenance
Faster Feedback
Integration with Other Tools
Focus on Business Requirements
Discover what are the different types of automation testing frameworks available and why to prefer the BDD framework over others here.
Framework Explanation:
The framework is simple because we included a feature file written in the Gherkin language, as you will notice. Basically, Gherkin is a simple plain text language with a simple structure. The feature file is easy to understand for a non-technical person and that is why we prefer the BDD framework for automation. To learn more about the Gherkin language please visit the official site here https://cucumber.io/docs/gherkin/reference/. Also, we have included the POST, GET, PUT & DELETE API methods. A feature file describes all these methods using simple and understandable language.
The next component of our framework is the step file. The feature and step files are the two main and most essential parts of the BDD framework. The step file contains the implementation of the steps mentioned in the feature file. It maps the respective steps from the feature file and executes the code.We use the behave library to achieve this. The behave understands the maps of the steps with the feature file steps as both steps have the same language structure.
Then there is the utility file which contains the methods which we can use more repeatedly. There is one configuration file where we store the commonly used data. Furthermore, to install all the dependencies, we have created a requirement.txt file which contains the packages with specific versions. To install the packages from the requirement.txt file we have the following command.
pip install -r requirement.txt
The above framework is explained in detail here. I suggest you please check out the blog first and understand the framework then we can move further with the workflow detail description. A proper understanding of the framework is essential to understand how to create the CI/CD workflow file.
How to create a Workflow File?
Create a GitHub repository for your framework
Push your framework to that repository
Click on the Action Button
Click on set workflow your self option
Give a proper name to the workflow file
“Additionally, please check out the below video for a detailed step understanding.” The video will show you how to create workflow files and the steps need to follow to do so.
github actions workflow file creation
Components of CI/CD Workflow File:
Events:
Events are responsible to trigger the CI/CD workflow file. They are nothing but the actions that happen in the repository for example pushing to the branch or creating a pull request. Please check the below sample events that trigger the CI/CD workflow file.
push: This event is triggered when someone pushes code to a branch in your repository.
pull_request: This event is triggered when someone opens a new pull request or updates an existing one.
schedule: This event is triggered on a schedule that you define in your workflow configuration file.
workflow_dispatch: This event allows you to manually trigger a workflow by clicking a button in the GitHub UI.
release: This event is triggered when a new release is created in your repository.
repository_dispatch: This event allows you to trigger a workflow using a custom webhook event.
page_build: This event is triggered when GitHub Pages are built or rebuilt.
issue_comment: This event is triggered when someone comments on an issue in your repository.
pull_request_review: This event is triggered when someone reviews a pull request in your repository.
push_tag: This event is triggered when someone pushes a tag to your repository.
To know more about the events that trigger workflows please check out the GitHub official documentation here
Jobs:
After setting up the events to trigger the workflow the next step is to set up the job for the workflow. The job consists of a set of steps that performs specific tasks. For every job, there is a separate runner or we can call it a virtual machine (VM) therefore each job can run parallelly. This allows us to execute multiple tasks concurrently.
A workflow can have more than one job with a unique name and set of steps that define the actions to perform. For example, we can use a job in the workflow file to build the project, test its functionality, and deploy it to a server. The defined jobs in the workflow file can be dependent on each other. Also, they can have different requirements than the others like specific operating systems, software dependencies or packages, or environment variables.
Discover more about using jobs in a workflow from GitHub’s official documentation here
Runners:
To execute the jobs we need runners. The runners in GitHub actions are nothing but virtual machines or physical servers. GitHub categorizes them into two parts named self-hosted or provided by GitHub. Moreover, the runners are responsible for running the steps described in the job.
The self-hosed runners allow us to execute the jobs on our own system or infrastructure for example our own physical servers, virtual machines, or containers. We use self-hosted runners when we need to run jobs on specialized hardware requirements that must be met.
GitHub-hosted runners are provided by GitHub itself and can be used for free by anyone. These runners are available in a variety of configurations. Furthermore, the best thing about GitHub-hosted runners is that they automatically update with the latest software updates and security patches.
Learn more about runners for GitHub actions workflow here from GitHub’s official documentation.
Steps:
Steps in the workflow file are used to carry out particular actions. Subsequently, after adding the runner to the workflow file, we define these steps with the help of the steps property in the workflow file. Additionally, the steps consist of actions and commands to perform on the build. For example, there are steps to download the dependencies, check out the build, run the test, upload the artifacts, etc.
Learn more about the steps used in the workflow file from GitHub’s official documentation here
Actions:
In the GitHub actions workflow file, we use actions that are reusable code modules that can be shared across different workflows and repositories. One or more steps are defined under actions to perform specific tasks such as running tests, building the project, or deploying the code. We can also define the input and output parameters to the actions which help us to receive and return the data from other steps in the workflow. Developers describe the actions, and they are available on GitHub Marketplace. To use an action in the workflow, we need to use the uses property.
Find out more about actions for GitHub actions from GitHub’s official documentation here
Now we have covered all the basic topics that we need to understand before creating our CI/CD workflow file for the API automation framework. Now, let’s start explaining the workflow file.
We use the name property to give the name to the workflow file. It is a good practice to give a proper name to your workflow file. Generally, the name is related to the feature or the repository name.
name: Python API CI/CD Pipeline
Event:
Now we have to set up the event that triggers the workflow file. In this workflow, I have added two events for your reference. The pipeline will trigger the push event for the ‘main‘ branch. Additionally, I added the scheduled event to automatically trigger the workflow as per the set schedule.
The above schedule indicates that the pipeline Runs at 12:00. Action schedules run at most every 5 minutes using UTC time.
We can customize the schedule timing as per our needs. Check out the following chron specification diagram to learn how to set the schedule timing.
Job:
The job we are setting here is to build. We want to build the project and perform the required tasks as we merge new code changes.
jobs:
build:
Runner:
The runner we are using here is a GitHub-hosted runner. In this workflow, we are using a Windows-latest virtual machine. The VM will build the project, and then it will execute the defined steps.
runs-on: windows-latest
Apart from Windows-latest, there are other runners too like ubuntu-latest, macos-latest, and self-hosted. The self-hosted runner is one that we can set up on our own infrastructure, such as our own server, or virtual machine, allowing us to have more control over the environment and resources.
Steps:
The steps are the description of what are the different actions required to perform on the project build. Here, the first action we are performing is to check out the repository so that it can have the latest build code.
steps:
- uses: actions/checkout@v3
Then we are setting up the Python. As this framework is an API automation testing framework using Python and Behave so we need Python to execute the tests.
- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: '3.8.9'
After we install Python, we also need to install the different packages required to run the API tests. Define these packages in the requirement.txt file, and we can install them using the following command.
As of now, we have installed all the packages and we can now run our API tests. We are running these tests with the help of the allure behave command so that once the execution is completed it will generate a Report_Json folder which is required to generate the HTML report.
- name: run test
run: behave Features -f allure_behave.formatter:AllureFormatter -o Report_Json
working-directory: .
continue-on-error: true
Here, we cannot share the generated Report_Json folder as a report. To generate the shareable report we need to convert the JSON folder to that of the HTML report.
Please find the attached GitHub repository link. I have uploaded the same project to this repository and also attached a Readme file that explains the framework and the different commands we have used so far in this project. Also, the workflow explanation is included for better understanding.
Conclusion:
In conclusion, creating a CI/CD pipeline workflow for your project using GitHub Actions streamlines the development and testing process by automating tasks such as building the project for new changes, testing the build, and deploying the code. This results in reduced time and minimized errors, ensuring that your software performance is at its best.
GitHub Actions provides a wide range of pre-built actions and the ability to create custom actions that suit your requirements. By following established practices and continuously iterating on workflows, you can ensure your software delivery is optimized and reliable.
I hope in this blog I have provided the answers to the most commonly asked question and I hope this will help you to start creating your CI/CD pipelines for your projects. Do check out the blogs on how to create a BDD framework for Web Automation and API automation for a better understanding of automation frameworks and how a robust framework can be created.
Static testing and dynamic testing are two essential approaches in software testing. While dynamic testing involves software execution, static testing examines work products, such as code or documentation, without execution. Static testing includes two methods: manual examination, where team members review work products to identify issues, and tool-driven evaluation, where specialized software tools perform static analysis.
Static testing is a critical technique as it can identify errors and defects in the software development lifecycle before software execution takes place. This can save time and resources by catching issues early, leading to improved software quality. Static testing also helps in improving the maintainability of the software and reducing the cost of testing by identifying issues early on. By detecting defects early, static testing can enhance the efficiency and effectiveness of the entire software development process.
What are Reviews in Software Testing:
In software testing, a review is a process of examining a software product or component by a group of individuals to detect defects, errors, or vulnerabilities. The primary goal of reviews is to identify and fix issues in the early stages of the software development life cycle to save time and cost.
Reviews play a crucial role in the software development process. It identifies and addresses errors and issues before they become problematic. Reviews can be informal or formal, depending on the level of rigor and structure involved.
Informal reviews are often less structured and involve team members informally reviewing code or other work products and discussing issues. On the other hand, formal reviews are more structured, involving documented procedures, and a diverse team with varied roles and perspectives, making them comprehensive.
Formal reviews typically produce documented results, including any issues identified and how they were resolved. This documentation is useful for tracking progress, ensuring accountability, and improving future reviews’ quality. Overall, reviews are essential to the software development process as they ensure the software is of high quality and meets the desired requirements.
Reasons why Reviews are Important:
First, they improve the quality of the software product or component by identifying and resolving defects early in the development process, preventing issues from reaching end-users, leading to enhanced customer satisfaction and reduced support requests.
Second, reviews reduce the overall cost and time of the software development project by catching defects early, saving time, and resources that would otherwise be needed to address issues in later stages of the development cycle.
Third, they provide an opportunity for knowledge sharing and collaboration among team members, promoting best practices and fostering improved teamwork and productivity.
Fourth, they ensure compliance with regulatory and industry standards by identifying issues that may be in violation of regulatory requirements or industry standards, allowing for corrective action to be taken before the product is released.
ISO/IEC 20246 is an internationally recognized standard that provides detailed guidelines for conducting reviews of work products during the software development process. The standard outlines team member roles and responsibilities and offers a range of review techniques to choose from, making the review process more efficient and effective.
The work product review process involves several main activities:
Planning :
During the planning phase, the following activities are performed:
Defining the scope
Estimating effort and timeframe
Identifying review characteristics
Selecting the people to participate in the review and allocating roles
Defining entry and exit criteria
Initiating the review:
During the initial review phase, the following activities are performed:
Distributing the work product and related material
Explaining the scope, objectives, process, roles, and work products to the participants
Answering any questions that participants may have about the review
Individual review:
During the individual review phase, participants perform the following activities:
Reviewing all or part of the work product
Noting potential defects, recommendations, and questions
Issue communication and analysis:
During the issue communication and analysis phase, the following activities are performed:
Communicating identified potential defects
Analyzing potential defects and assigning ownership and status to them
Evaluating and documenting quality characteristics
Evaluating the review findings against the exit criteria to make a review decision
Fixing and reporting:
During the fixing and reporting phase, the following activities are performed:
Creating defect reports for findings that require changes to a work product
Fixing defects found in the work product reviewed
Communicating defects to the appropriate person or team and recording the updated status of defects
Gathering metrics
Checking those exit criteria are met
Accepting the work product when the exit criteria are reached
In a formal review, there are several roles and responsibilities that are typically assigned to participants.
Roles and Responsibilities:
Moderator/Chairperson: Responsible for conducting the review meeting. It ensures that the review is conducted according to the review process, and ensures that the review objectives are met.
Author: The person who wrote the work product being reviewed.
Reviewer: A person who is responsible for reading and analyzing the work product being reviewed and identifying any defects, issues, or potential improvements.
Recorder/Scribe: Responsible for recording the minutes of the review meeting and documenting the issues raised, the decisions made, and any action items assigned.
Manager: A person who is responsible for managing the review process and ensuring that the review is conducted effectively.
Quality Assurance Representative: Responsible for ensuring that the review process is in compliance with organizational quality standards and procedures.
Technical Expert: A person who has expertise in the subject matter being reviewed. Also, the person who can provide technical advice and guidance during the review process.
Each of these roles has specific responsibilities and tasks that are assigned to them during the review process. So to ensure a successful review, it is crucial to clearly communicate the roles, responsibilities, processes, and objectives to each participant. This will ensure everyone is aligned and comprehends their expected roles during the review.
Types of reviews include:
Additionally, there are several types of reviews that can be conducted during the software development lifecycle
Peer Review: A review of a work product involves one or more peers with comparable expertise and authority level.
Technical Review: A review that focuses on the technical aspects of a work product, such as design, code, or architecture.
Walkthrough: A review that involves the author of the work product presenting the work product to a group of reviewers, who ask questions and provide feedback.
Inspection: A highly formal and structured review that involves a team of reviewers examining a work product in detail to identify defects and potential improvements.
Audit: A review that focuses on ensuring compliance with standards, regulations, or policies.
Management Review: A review that is conducted by management to ensure that the project or work product is progressing as planned and to identify any potential issues or risks.
Review type choice depends on factors like work product nature, review goals, and available resources.
Conclusion:
Reviews are a critical component of the software development lifecycle. By tapping into the collective knowledge and expertise of team members, reviews enable the early identification of defects and potential improvements, leading to improved software quality and reduced costs and time. Furthermore, reviews facilitate communication and collaboration among team members, fostering a culture of continuous improvement and knowledge sharing.
In addition, reviews help organizations to comply with regulatory and industry standards, thereby mitigating risks and maintaining the trust of their stakeholders. Overall, reviews are a valuable tool for any software development team aiming to deliver high-quality software products efficiently and effectively.
Apurva is a Test Engineer, with 3+ years of experience in Manual and automation testing. Having hands-on experience in testing Web as well as Mobile applications in Selenium and Playwright BDD with Java. And loves to explore and learn new tools and technologies.
To ensure comprehensive mobile app testing, it is essential to do 360-degree testing for functions, appearance, and performance on multiple devices with various configurations, sizes, resolutions, and platforms.
There are three ways to conduct mobile app testing:
Emulators for Android devices and Simulators for iOS devices
Real physical devices
Cloud-based real devices platform
Testing an application on different platforms, including emulators and simulators, as well as physical devices, comes with a set of challenges.
Emulators and Simulators:
When it comes to emulators and simulators, creating multiple virtual devices with different configurations can be difficult, and they may not accurately replicate real-world situations compared to physical devices.
Physical devices:
On the other hand, testing on physical devices poses its own set of difficulties. It requires a large number of devices to test app compatibility with different configurations, and the list of devices keeps changing as new models are introduced to the market. Additionally, cross-location testing can be challenging since carrying multiple devices across different locations can be cumbersome.
To address these challenges, a possible solution is to use cloud-based platforms for testing applications in real-world conditions. By doing so, we can perform robust testing with a variety of devices and configurations without the need to physically acquire and maintain them.
Cloud-based platform (BrowserStack) :
BrowserStack is a cloud-based real devices platform that provides support for both manual and automated testing of mobile apps for both Android and iOS devices.
One of its standout features is the App Live feature, which allows users to manually test their mobile apps on over 3000 real Android and iOS devices.
BrowserStack also supports 15+ native features, such as GPS, network simulation, and localization, which can be used to test mobile apps in real-world scenarios.
Other features include debugging apps, viewing crash reports and logs, inspecting UI elements, and using stack traces to find and fix bugs instantly. These features make it easy for developers to identify and fix issues quickly and efficiently.
Furthermore, BrowserStack supports testing across different environments, including Dev, QA, Staging, and Production apps from the play store or app store. This makes it easy for developers to test their apps in various environments and ensure that they are working correctly in each environment.
To test a mobile app on BrowserStack, follow these steps:
4. Here in the left panel “SELECT SOURCE” you can see different options available to upload or install apps.
i. Uploaded Apps – You can upload an apk or ipa file here.
ii. Sync with App Centre – If you have access to the app center you can use this option to directly install the app from App Centre.
iii. Install via TestFlight – While testing on iOS devices you can install the app directly from TestFlight also.
iv. Install via Play Store – Select a device, sign in to the play store and install the app if available on Play Store.
v. Install via App Store – Select a device, sign in to the App store and install an app, if available on App Store.
Now let’s explore how to upload apk file from a local machine using Upload Apps here…
i. Click on Upload – Select apk/ipa file from your local machine to be uploaded.
ii. Select the uploaded apk/ipa file and device from the list of available devices in the right panel.
iii. You can select any device from the list for the selected device category.
iv. Once you click on the device name the device will be launched and the app will be installed on it.
Now once the app is launched on the selected device you can start testing the app.
Let’s explore various features that can be used effectively while testing…
1. Report Bug:
If you find a bug while testing, BrowserStack provides the option to highlight the bug on the screen and add a bug description.
You can download the image to include it in the defect report later or send a screenshot to the developer or your team via email.
Additionally, you can share the image using various applications such as Jira, Slack, GitHub, etc. This option is visible in the screenshot below.
Then click on Cancel to return back to the testing screen.
2. Record Session:
This feature can capture a video recording of a sequence of events that led to a bug.
Choose the screen to be recorded.
After recording a session using BrowserStack, you can stop the sharing by clicking on the “Stop Sharing” button. Then, you can download the recording for later use.
3. Switch Device
This feature allows users to switch to another device at any point in time to test the same app on different devices.
The toolbar on the right side of BrowserStack provides various features to simulate real device features, including:
View open apps on the device.
Kill existing apps if they become unresponsive or test app behavior on a relaunch.
Capture screenshots, similar to taking screenshots on a physical phone.
Rotate the screen to test the app’s responsiveness to different screen orientations.
Some other important features are:
Change Location: This feature is used to test app behavior and response across different geographical locations. You can use the “Device GPS” option to set the exact latitude and longitude coordinates or the “IP Location (Beta)” option to select a specific region from a list of locations. See the screenshot below for reference.
Change Language – This feature can be used to test apps in different languages.
File & Media – This feature allows you to transfer various types of files to and from the device during testing.
On the right-hand side of the window, we have the following options in the Dev Tools section:
INSPECT (BETA) –This option is used to find element locators for app automation testing.
NETWORK Tab – This feature is useful for monitoring backend HTTP requests and responses in real-time. It can help you to debug the cause of failures during certain events in the app.
On the BrowserStack cloud platform, you can also test various native device features just like physical devices. These include:
Geolocation testing
Push notifications
Pre-loaded images
Network simulation
Testing dependent apps
Uninstalling apps
Conclusion:
Overall, BrowserStack is a convenient and easy-to-use cloud-based real devices platform that provides a wide range of features for mobile app testing. Its support for manual and automated testing, along with its native features and ability to test across different environments, makes it a popular choice among developers.
Manisha is a Lead SDET at SpurQLabs with overall experience of 3.5 years in UI Test Automation, Mobile test Automation, Manual testing, database testing, API testing and CI/CD. Proven expertise in creating and maintaining test automation frameworks for Mobile, Web and Rest API in Java, C#, Python and JavaScript.
API’s the term we heard a lot and wanted to know more about it. The questions that come to our mind are what is it? Why is it so important? How to test it? So, let’s just explore these questions one by one. API testing is accessible only if you know what to test and how to test. Again, a proper framework will help you to achieve your goals and deliver a good quality of work. The importance of automation framework and the factors we should consider for choosing the proper framework are described in our previous blog. Please go through the blog here, then you can start reading this blog because you will have a good understanding of automation testing frameworks.
To build the API testing framework we will be using the BDD approach. Again, why I have chosen a BDD framework for API testing the reason is very simple the BDD approach provides you with an easy understanding of the framework, you can easily maintain the framework and they have feature files that are very easy to understand for a non-technical person.
What is API?
API (Application Programming Interface) is like a mechanism that works between the two software components and helps them to communicate with each other. The communication happened using sets of definitions and set protocols. In simple language, API works as an intermediate between two systems and helps them exchange data and communicate. The working mechanism of Rest API is straightforward they work by sending requests and receiving a response in a standardized format. Usually, the standardized format used for Rest API is JSON i.e. (JavaScript Object Notation)
Let’s understand it better with an example. Consider you are using a ticket booking app to book a flight ticket. As the app is connected to the internet so it will set data to the server. When the server receives the data it interprets it and performs the required actions and sends it back to your device. Then the application translates that data and display the information in a readable way. So this is how API works. I hope you have understood the working mechanism of API’s now let’s discuss the next topic i.e.
What is API Testing?
As we have understood what is an API and how they work so let’s see why their testing is important. Basically, API testing is a part of software testing that includes the testing of the functionality, reliability, security, and performance of API. API is used for data transfer and to establish communication between the two systems so testing APIs includes verifying that the APIs are meeting its requirement, performing as per the expectations, and can handle a variety of inputs. This testing provides you the information that the API’s functionality is correct and efficient and the data they return is accurate and consistent.
Why is API Testing Important?
API testing is an important part of a Software testing process as it helps you to understand the functionality of the working APIs and validate any defect present before the application is released to the end users. The other key reasons why API testing is important to include:
Ensuring Functionality
Validating data integrity
Enhancing the Security
Improving the Performance
Detecting Bugs and Issues
Improving readability and stability
Facilitating integration and collaboration
All the above-mentioned points get checked and validate in API testing. Till now we have discussed what is api, what is api testing, and why it is important. Let’s see what different tools are available to conduct the manual as well as automation testing of API.
Tools for Manual API Testing:
Postman
SoapUI
Insomnia
Paw
Advanced REST Client (ARC)
Fiddler
cURL
Tools for API Automation Testing:
Postman
SoapUI
RestAssured
RestSharp
Apache HTTP client
JMeter
Karate
Newman
Pact.js
Cypress.js
These are just a few examples of the tools available for both manuals as well as automation testing of API. Each mentioned tool has its own strength and weakness and the choice of the right tool for your API testing depends upon the requirement and the specific needs of the project. These tools will help us to ensure that the APIs meet the desired functionality and performance requirements.
Now we are more familiar with APIs so let’s start the main topic of our discussion and i.e. Python Behave API Testing BDD Framework.
Framework Overview:
To validate all the above-mentioned points creating a robust API testing framework is very essential. With the help of the below-mentioned steps, you will come to know how to create your own API testing framework. Here, we are going to create a BDD framework. Please go through this blog before starting to read this blog as the previous blog will help you to understand the advantages of BDD and this blog is linked to the previous blog topics. You can read the previous blog here.
This framework structure contains a feature file, a step file, and a utility file. We will be discussing all these terms shortly. To create such a framework you need to follow certain steps to make your work tedious-free and easy.
Install all the required packages using the below command as long as you have all the packages mentioned in rquirement.txt with the right version number
We can also install the mentioned packages from the settings of Pycharm IDE
Step2: Creating Project
After understanding the prerequisites the next step is to create a project in our IDE. Here I am using a Pycharm Professional IDE. As mentioned in the above step, we will install the packages mentioned in the requirement.txt file. Please note it is not compulsory to use Pycharm Professional IDE to create this framework you can use the community version too.
Step3: Creating a Feature File
In this, we will be creating a feature file. A feature file consists of steps. These steps are mentioned in the gherkin language. The feature is easy to understand and can be written in the English language so that a non-technical person can understand the flow of the test scenario. In this framework we will be automating the four basic API request methods i.e. POST, PUT, GET and DELETE. We are taking https://reqres.in/
We can assign tags to our scenarios mentioned in the feature file to run particular test scenarios based on the requirement. The key point you must notice here is the feature file should end with .feature extension. We will be creating four different scenarios for the four different API methods.
Feature: User API
Verify the GET PUT POST DELETE methods of User API
@api
Scenario: Verify GET call for single user
When User sends "GET" call to endpoint "api/users/2"
Then User verifies the status code is "200"
And User verifies GET response contains following information
| First_name | Last_name | Mail-id |
| Janet | Weaver | janet.weaver@reqres.in |
@api
Scenario: Verify POST call for single user
When User sends "POST" call to endpoint "api/users"
| Name | Job |
| Yogesh | SDET |
Then User verifies the status code is "201"
And User verifies POST response body contains following information
| Name | Job |
| Yogesh | SDET |
@api
Scenario: Verify PUT call for single user
When User sends "PUT" call to endpoint "api/users/2"
| Name | Job |
| Yogesh | SDET |
Then User verifies the status code is "200"
And User verifies PUT response body contains following information
| Name | Job |
| Yogesh | SDET |
@api
Scenario: Verify DELETE call for single user
When User sends DELETE call to the endpoint "api/users/2"
Then User verifies the status code is "200"
Step4: Creating a Step File
Unlike the automation framework which we have built in the previous blog, we will be creating a single-step file for all the feature files. In the BDD framework, the step files are used to map and implement the steps described in the feature file. Python’s behave library is very accurate to map the steps with the steps described in the feature file. We will be describing the same steps in the step file as they have described in the feature file so that behave will come to know the step implementation for the particular steps present in the feature file.
from behave import *
from Utility.API_Utility import API_Utility
api_util = API_Utility()
@when('User sends "{method}" call to endpoint "{endpoint}"')
def step_impl(context, method, endpoint):
global response
response = api_util.Method_Call(context.table, method, endpoint)
@then('User verifies the status code is "{status_code}"')
def step_impl(context, status_code):
actual_status_code = response.status_code
assert actual_status_code == int(status_code)
@step("User verifies GET response contains following information")
def step_impl(context):
api_util.Verify_GET(context.table)
response_body = response.json()
assert response_body['data']['first_name'] == context.table[0][0]
assert response_body['data']['last_name'] == context.table[0][1]
assert response_body['data']['email'] == context.table[0][2]
@step("User verifies POST response body contains following information")
def step_impl(context):
api_util.Verify_POST(context.table)
response_body = response.json()
assert response_body['name'] == context.table[0][0]
assert response_body['job'] == context.table[0][1]
@step("User verifies PUT response body contains following information")
def step_impl(context):
api_util.Verify_PUT(context.table)
response_body = response.json()
assert response_body['Name'] == context.table[0][0]
assert response_body['Job'] == context.table[0][1]
@when('User sends DELETE call to the endpoint "{endpoint}"')
def step_impl(context, endpoint):
api_util.Delete_Call(endpoint)
Step5: Creating Utility File
Till now we have successfully created a feature file and a step file now in this step we will be creating a utility file. Generally, in Web automation, we have page files that contain the locators and the actions to perform on the web elements but in this framework, we will be creating a single utility file just like the step file. The utility file contains the API methods and the endpoints to perform the specific action like, POST, PUT, GET, or DELETE. The request body i.e. payload and the response body will be captured using the methods present in the utility file. So the reason these methods are created in the utility file is that we can use them multiple times and don’t have to create the same method over and over again.
import json
import requests
class API_Utility:
data = json.load(open("Resources/config.json"))
api_url = data["APIURL"]
global response
def Method_Call(self, table, method, endpoint):
if method == 'GET':
uri = self.api_url + endpoint
response = requests.request("GET", uri)
return response
if method == 'POST':
uri = self.api_url + endpoint
payload = {
"name": table[0][0],
"job": table[0][1]
}
response = requests.request("POST", uri, data=payload)
return response
if method == 'PUT':
uri = self.api_url + endpoint
reqbody = {
"Name": table[0][0],
"Job": table[0][1]
}
response = requests.request("PUT", uri, data=reqbody)
return response
def Get_Status_Code(self):
status_code = response.status_code
return status_code
def Verify_GET(self, table):
for row in table:
first_name = row['First_name']
last_name = row['Last_name']
email = row['Mail-id']
return first_name, last_name, email
def Verify_POST(self, table):
for row in table:
name = row['Name']
job = row['Job']
return name, job
#Following method can be merged with POST, however for simplicity I kept it
def Verify_PUT(self, table):
for row in table:
name = row['Name']
job = row['Job']
return name, job
def Delete_Call(self, endpoint):
uri = self.api_url + endpoint
response = requests.request("DELETE", uri)
return response
Step6: Create a Config file
A good tester is one who knows the use and importance of config files. In this framework, we are also going to use the config file. Here, we are just going to put the base URL in this config file and will be using the same in the utility file over and over again. The config file contains more data than just of base URL when you start exploring the framework and start automating the new endpoints then at some point, you will realize that some data can be added to the config file.
Additionally, the purpose of the config files is to make tests more maintainable and reusable. Another benefit of a config file is that it makes the code more modular and easier to understand as all the configuration settings are stored in a separate file and it makes it easier to update the configuration settings for all the tests at once.
"APIURL": "https://reqres.in/"
Step7: Execute and Generate Allure Report
The reason behind using allure reports as a reporting medium is because the allure report provides detailed information about the test execution process and results which includes the test status, test steps, duration, and screenshots of the test run. The report is generated in HTML i.e. web format making it easy to share with team members and with clients and easy to understand. It provides a dashboard that is user-friendly having interactive charts and graphs that provide a detailed analysis of the test results.
Let’s understand how to execute API tests and generate an allure report for automated API calls. To generate the report we will have to execute the test using the terminal or command line. There are two steps to follow sequentially they are as follows:
The purpose of the above command is to execute the test present in the mentioned feature file and generate a JSON report folder.
allure generate Report_Json -o Report_Html –clean
This command is used to generate an HTML report from the JSON report. So, that it is easy to understand and can be shared with team members or clients.
Please find the attached GitHub repository link. I have uploaded the same project to this repository and also attached a Readme.md file which explains the framework and the different commands we have used so far in this project.
Before creating a framework it is very important to understand the concept and I hope I have provided enough information for the different queries on APIs. In conclusion, creating a BDD API testing framework using Python and Behave is easy to process if you know how to proceed further. By following the steps outlined in this blog I am sure you can create a powerful and flexible framework that will help you to define and execute the test cases, generate a detailed report with allure and also iterate with other testing tools and systems. Again I am suggesting you check out the previous blog here because that will clear most of your doubts on automation testing frameworks and will help you to create your own automation testing framework.
