The World of Automation Testing
Quality assurance is an important step in software development. It has fuelled growth and innovation, resulting in more efficient and accurate testing environments. From the initial days of manual testing – which involved manual selection of test cases, conducting the tests, recording of results and deriving conclusions – we’ve moved to the age of scripted and data-driven automation, where repeatable and consistent testing can be done driven by a wide range of data. Even with all this growth and evolution, the presence of unbiased and widely varying data with brand new scenarios in mind are hard to find. This is compounded by the additional amount of manual input required for testing, scripting and other processes.
The Advent of Generative AI
Generative AI is a powerful tool capable of revolutionizing most fields including software testing. Software testing requires the existence of robust data that can cover most testing scenarios and edge cases along with code that is well maintained to run all the test scenarios. Generative AI can be used to generate data and scripts by understanding patterns and user/stakeholder requirements.
How Would a Solution of Generative AI in Testing Work?
Generative AI can be implemented in various ways in testing like:
- Generating Test Data
Having effective data that correctly tackle edge cases and finding possible vulnerabilities during testing is always hard. This is where Generative AI can help out. Generative AI can be utilized to generate diverse test data that covers all possible scenarios. This process is pretty cumbersome to be done manually. Such generated data can then be combined with information about usage patterns and application requirements to create effective test cases that can better detect vulnerabilities.
- Generation of Test Scripts
Creation and maintenance of test scripts is one of the most resource-intensive tasks in automated testing. It requires considerable time and effort from a qualified group of individuals. It also limits the access of various stakeholders in generating effective and important test scenarios. With Generative AI, such individuals can give accurate descriptions of the testing to be performed while the AI can handle the job of generating efficient and accurate test scripts. The individuals once involved in creating and maintaining test scripts can now move to assisting the AI and helping out in getting a better understanding of the scripts/results generated.
- Bug Detection
Generative AI can also be used to analyze previous testing patterns and occurrence of bugs to identify possible points of failures. This information can then be used to create new test cases tackling those exact scenarios. Bug detention and faulty area detection can be addressed.
These are just a few of the many ways in which Generative AI can help out in automation testing. Scaling test cases along with the core software, test code maintenance, consistent test quality etc. are other prominent benefits of using Generative AI for testing.
Challenges in Implementing Generative AI
Using Generative AI has its own set of challenges though. We will list them here.
- Inability to generate meaningful test data/tests due to lack of understanding of the scenario at hand.
- Generative AI models like GANs, large transformers etc. require huge amounts of compute resources, which can pose challenges for businesses.
- Training/fine-tuning the Generative AI model requires large amounts of quality data to work with.
- Interpreting AI-generated tests, data etc. require additional skilled individuals especially in cases of failures during testing.
Generation of Synthetic Test Data Using Generative AI
Synthetic test data is not mock data. Synthetic test data, created by Gen AI, which has been trained on real data, maintains the structure and referential integrity of the original data while supporting relational structures.
This AI-generated synthetic data is not simply mock or fake data; it accurately reflects customer behavior as observed in production data. Unlike manually generated data, it is produced by a powerful AI engine that comprehensively learns the characteristics of its training dataset, achieving 100% test coverage.
High-quality synthetic data generators can efficiently automate the process of test data generation, mitigating privacy concerns. It is crucial to utilize customer data in its synthetic form to uphold privacy standards while preserving the business rules inherent in the data.
Here’s the rough idea on how to do this:
The rough idea is actually very simple:
- Prompt a local running Llama2, Falcon or Mistral instance. Give it instructions regarding which type of request is expected (valid/non-valid request, XSS attack …).
- Execute the requests.
We have written extensively on how to host your own local LLM on E2E Cloud in several blogs. As a first step, launch a GPU node to host your own LLM.
We recommend using your own privately hosted LLM because it keeps your data safe, and you avoid incurring API costs in the long run with the proprietary ones (as your token sizes increase, so does your cost).
So go ahead and register with E2E Cloud and spin up a GPU node.
Next, you can follow the guide here to deploy Llama2.
We also recommend using Visual Studio with Remote Explorer extension, to use the remote machine almost as a local development environment.
Once you are connected and have your LLM running, try out a prompt like this:
Generation of Scripts That Generate Test Data
While the above is one approach, the challenge you will face is that this will generate a limited amount of data.
So, how do you solve this? Simple - use an LLM to generate a Python script, which then generates the test data. For this, it would be better if you use a code-tuned LLM. One such AI coding assistant is StarCoder.
StarCoder is an LLM model for code, designed by Hugging Face, and trained on permissively licensed data from GitHub. It has been trained on over 80 programming languages, Git commits, GitHub issues, and Jupyter notebooks.
The model has approximately 15 billion parameters trained on over 1 trillion tokens. StarCoder and its variants, StarCoderBase and StarCoderPlus, are aimed at code generation tasks across a wide range of programming languages.
StarCoderEx is a tool available for Visual Studio Code, created to generate code using StarCoder, and is positioned as an alternative to GitHub Copilot.
Follow the steps here to deploy StarCoder. Alternatively, you could deploy Code-Llama.
Then prompt it like this:
This will generate a Python script that you can then use for synthetic test data generation.
Generation of Test Scripts
One of the most exciting applications of AI coding assistants is generating test scripts. AI coding assistants like StarCoder or Code-Llama can help any developer create unit tests for their Python code, without compromising company IP or customer data.
What you need to do is provide a brief description of the class and its purpose, the methods you want to test, and any dependencies the class might have on external resources or other classes.
They can then be used to automatically generate unit tests using AI and metaprogramming techniques, ensuring comprehensive test coverage for your code. This feature can save a lot of time and effort, as writing test scripts can be a tedious and time-consuming task.
Since you now have an AI-Coding LLM running, we can leverage that to generate test scripts as well. Here’s how you prompt that.
Try this out, and play around with the prompt. This will give you a quick way to generate test scripts that you can then execute and test your code on. You can prompt it to vary the test scenarios.
Here’s a Python code to simplify this further (modify to use the right LLM function in your case):
Coding Assistants for Bug Detection
It doesn’t stop there. AI coding assistants can actually be incorporated in a lot of more advanced ways – for example, to improve testing processes through bug detection, analysis, code reviews and more.
AI coding assistants can significantly expedite and enhance the bug detection process in software development in several ways:
Analysis
AI can perform static analysis to scrutinize code without executing it to find common bugs, coding standard violations, or other potential issues. It can identify patterns associated with bugs and suggest corrections. This is becoming a potent approach in ensuring bugs are detected before even the code is pushed.
Code Review Automation
AI can automate code reviews, checking for bugs, inconsistencies, or violations of coding standards. It can also help in identifying sections of code that are likely to cause bugs based on historical data, and suggest better coding practices.
Automated Testing
As we saw above, you can use AI to generate and execute tests at scale to uncover bugs. It can also identify areas of the code that are under-tested and suggest additional tests. This is an area where more development is expected to happen in the near future.
Anomaly Detection
AI can also be used to analyze historical data and understand normal code behavior. This can help identify anomalies and point to issues with the code.
Natural Language Processing (NLP)
Utilizing LLM-NLP, AI models can process bug reports, extract relevant information, and even link bugs to code, helping to speed up the bug-fixing process. For instance, you can use natural language to talk to your code, and understand its flow and how it's working. For companies which have a ton of legacy code, this can prove to be extremely useful.
These features not only enhance bug detection but also contribute to a more efficient and effective software development process, saving time and ensuring higher code quality.
Conclusion
The advent of Generative AI has been revolutionary in many areas. Software quality testing has always been a continuously evolving field throughout its existence. Now as one enters the age of Gen AI, the possibilities are endless with huge leaps in automation testing to be expected.