Blog post
In recent years, Artificial Intelligence (AI) has made significant strides in its ability to produce software requirements specifications. One of the most impressive advancements has been the GPT-3, an AI-powered language model developed by OpenAI. In this blog post, we will discuss how capable the GPT 3 is in helping produce software requirements specifications, particularly when it comes to formal requirements formats such as use cases, acceptance criteria, and user stories. We will also assess the potential benefits and drawbacks of using GPT-3 for this purpose.
I didn’t write that introduction you’ve just read.
My personal BA assistant did when I told it to “Write a short introduction to a blog post discussing how capable the GPT 3 is in helping produce software requirements specifications, especially when it comes to formal requirements formats such as use cases, acceptance criteria, and user stories.”
Admittedly, the result is not perfect, the generated text is somewhat generic and generally lacks style but contextually, it does fit with the rest of this text. Overall, the wording and phrasing are decent, and all things considered, the result is of a quality that could be expected from an intern or a junior when given the same request as stated above.
A “proper” Introduction
At this point, most of you’ve had probably at least heard about OpenAI’s GPT-3, the AI language model which is capable of understanding prompts written in regular, conversational English, and synthesizing meaningful responses to them. Also, quite a few of you have already tried it — either through OpenAI’s GPT-3 playground or via awesome GPTChat at chat.openai.com.
The consensus of those who did try it (including me) seems to be that they found GPT-3 to be something that was firmly in the realms of science fiction just a couple of years ago.
The topic of this post, however, is not whether GPT-3 is awesome or not, (since that would be a rather boring two-sentence article ending with a “YES, it certainly is”), but rather how we could utilize this new and seemingly incredibly sophisticated tool that somehow landed into our laps rather unexpectedly.
And since I’m a business analyst and a functional designer, we’ll focus on utilizing the AI model for tasks that I care about the most — namely writing software requirements.
Talking with the machine
The GPT-3 (Generative Pre-trained Transformer) is an advanced AI language model (neural network) which has been trained through countless *millions* of texts from all over the internet (and beyond), including Wikipedia pages, and books on all conceivable subjects. What GPT-3 does is it takes a textual query written in plain English (a.k.a. the “prompt”) and then produces a string of words that it thinks will best fit and complete the original input query.
The effect is that the interaction with the system looks and feels like a real conversation — although fundamentally that is not the case.
The interesting feature of the model, at least for us, is that for it to be able to provide meaningful “responses” that fit the context of the prompt, rather than just creating a string of grammatically correct, but essentially meaningless sequences of words, the AI model also had to develop an abstract understanding of all the concepts described in those millions of texts that it was trained upon.
And since those millions of texts included a LOT of texts discussing (and containing) use cases, acceptance criteria, user stories, and several other requirements-related topics, as a consequence, the model developed an “understanding” of those concepts as a result of it’s learning process.
That in effect means that (as long as we’re willing to accept some “fuzziness” in the answers we get to our prompts, and understand that they might not be fully accurate regardless of how confident and definitive they look) we should be able to use the GPT-3 to produce those formats by simply asking it to do so in plain English.
So I tried to do just that, and this post resulted from the mind-blowing results that I got, and in the remainder of this chapter, I’ll share the results of my “conversations” and as a result, hopefully, you’ll get at least a bit of your mind blown too.
I used the OpenAI playground (https://beta.openai.com/playground) for interaction with the GPT-3 and my prompts (a.k.a. questions or queries) will be styled as bold and italicized text and copy-pasted responses I’ve got from the system will be italicized directly below them. All prompts and responses are pasted exactly as the GPT-3 AI model delivered them on screen without any change or intervention.
Finally, for the model settings, I’ve used the default values (text-davinci-003 model, the temperature at 0.7, and response length at 256 tokens, etc.)
With all that out of the way, let’s start and see what this thing has to say.
1. Summaries
I asked for a summary first — something I’d typically write at the beginning of any SRS that I write as a business analyst.
Write a 200-word summary depicting the standard classifieds system, such as craigslist.
Classifieds systems, such as craigslist, are online marketplaces where individuals can buy, sell, trade and exchange services and goods. These systems are usually free to use and are highly accessible, allowing users to post listings in various categories and to search for items they might be interested in.
When using a classifieds system, users typically start by creating an account and providing some basic details. After they have completed the registration process, they can begin to post their items for sale or search for items they would like to buy. Users can also browse through the different categories, such as cars, electronics, furniture, and services.
Once a user finds an item they are interested in, they can contact the seller directly and arrange for a purchase or exchange. The seller and buyer will then agree on a price and payment method, and the seller will ship the item or provide the service. If the item is a physical item, buyers may be able to inspect it before buying.
Overall, classifieds systems such as craigslist provide a quick and easy way for individuals to buy, sell, and trade items and services. By using these systems, buyers and sellers can connect with each other directly and efficiently. As a result, classified
Well, that’s impressive to say the least, even with the abrupt mid-sentence ending which was brought about by my insisting that the result should be 200 words long — as that kind of specific “quantified” requests tends to produce somewhat weird results and are generally “frowned upon” by most AI models today.
Let’s try to remove this hard “200 words” limit and see what we’ll get then…
Write a summary depicting the standard classifieds system, such as craigslist.
The classifieds system is an online marketplace for individuals to buy, sell, and trade goods and services. It is a platform for users to post classified ads, with categories ranging from jobs, housing, and items for sale. Users can search for listings by geographic location, type of product, or keyword. Items can be posted for free or for a fee. The listings typically include detailed descriptions of the item, along with photos, price, contact information, and other relevant details. Buyers and sellers can communicate through the website messaging system, allowing them to negotiate terms and arrange for payment and shipping. The classifieds system provides a safe and convenient way for individuals to buy, sell, and trade products and services.
That’s more like it. So it seems that it’s able to produce pretty decent summaries, but that’s not surprising as that’s about the first thing that we’d expect from the system that is trained to understand and construct a meaningful natural language output in the first place.
But let’s move from the “easy stuff” and go for something more specific and formal this time. Let’s try use cases.
2. Use cases
Write a technical use case describing purchasing a subscription for a given service provided by a system-under-design, where the subscription management is performed using a Stripe payment processor.
Use Case Title: Purchase Subscription Using Stripe
Actor: End User
Pre-conditions:
– End User is logged into the system
– End User has valid credit card details
Main Flow:
1. End User selects the desired subscription level
2. System displays the associated subscription fee
3. End User confirms the subscription fee
4. System redirects End User to Stripe payment processor
5. End User enters the credit card details
6. Stripe payment processor verifies the credit card details
7. Stripe payment processor communicates the result of the verification back to the System
8. If verification is successful, System creates a subscription for End User
9. System sends a confirmation to End User
Post-conditions:
– End User has a valid subscription to the service
– Stripe payment processor has charged the credit card for the associated subscription fee
Wow. Although this is not exactly the style of use cases I use, it’s a decent use case nonetheless, and built from a prompt that could as easily be given to your human junior colleague, and the result is quite on the level that you’d expect the intern or a junior to produce. Impressive.
But let’s try to be more specific and demand a use case format that I DO prefer. That should make things quite a bit more complicated for the AI.
Write a use case for a registration process that uses the already existing google or Facebook accounts (a.k.a. “register with google” or “register via Facebook”). Use the writing style depicted in the book “Writing effective use cases” by Alistair Cockburn. Include all relevant extensions of a produced main success scenario.
Title: Register with Google or Facebook
Primary Actor: User
Goal in Context: The user is able to sign up for an account using their existing Google or Facebook account.
Stakeholders:
User: To create an account with minimal effort.
System: To securely store the user information.
Preconditions:
The user has a valid Google or Facebook account.
Main Success Scenario:
1. User clicks on “Register with Google” or “Register via Facebook”.
2. System authenticates the user’s Google or Facebook account.
3. System displays a form with the user’s information pre-filled.
4. User verifies and updates the information if needed, then clicks “Submit”.
5. System stores the updated information and creates a user account.
Extensions:
4a. User does not want to use their existing information.
4a1. System displays an empty form.
4a2. User fills out the form and clicks “Submit”.
5a. System is not able to store the user information.
5a1. System displays an error message.
Ok, well, this was the moment when my eyes started popping out. The style is not 100% right, but it’s almost there and all the important parts I’ve asked for are included. Additionally, it seems to know what the extensions are, how they are marked and how their steps are numbered.
So it seems that the use cases are not a problem for our AI-powered BA assistant, let’s try acceptance criteria.
Acceptance criteria
Write a comprehensive set of acceptance criteria describing all behaviors for the following user story: “As a visitor, I want to be able to register using my existing google account”
Acceptance Criteria:
1. The user should be able to click a “Register using Google” button on the registration page
2. The user should be redirected to a Google login page
3. The user should be able to enter their Google credentials to log in
4. The user should be provided feedback if the login is successful or unsuccessful
5. If the login is successful, the user should be redirected back to the registration page with their details pre-filled
6. The user should be able to submit the form to complete the registration process
7. The user should be provided with a confirmation message upon successful registration
8. The user should be able to log in with their Google account in the future
Not bad. The AI seems to understand the concept of user stories and what they represent, interpret that particular user story and produce a fairly complete set of acceptance criteria. I don’t like the style though — I prefer that acceptance criteria be either “present tense statements that can easily be demonstrated to be true or false” or gherkin. Let’s ask for those and see what we get.
We’ll try present tense statements first.
Write a comprehensive set of acceptance criteria describing all behaviors for the following user story: “As a visitor, I want to be able to register using my existing google account”. Use present tense statements that could easily be demonstrated (to a person) to be either true or false for all acceptance criteria you’ll produce.
1. I can see a ‘Sign up with Google’ button on the registration page.
2. When I click on the ‘Sign up with Google’ button, a pop-up window appears.
3. The pop-up window contains a Google sign in form.
4. I can enter my Google account credentials into the form.
5. I can see a ‘Sign in’ button on the form.
6. When I click on the ‘Sign in’ button, I am successfully logged in.
7. I am presented with a confirmation of successful registration.
Again, wow. Still, there are minor things I see here, and I’d probably include a few more acceptance criteria for this story, but this is really “serviceable” to say the least. Well done machine!
Let’s see how it’ll fare with gherkin — and by now, I do have a sneaking suspicion it will do well.
Write a comprehensive set of acceptance criteria describing all behaviors for the following user story: “As a visitor, I want to be able to register using my existing google account”. Use gherkin format.
Given a user is a visitor
When the user clicks on the “Register” button on the home page
Then the user is presented with a Google login form
Given the user has an existing Google account
When the user enters their Google credentials
Then the user is logged in to the site
Given the user is now logged in to the site
When the user enters their email address
Then the user is asked to enter a password
Given the user has entered a password
When the user clicks “Submit”
Then the user is registered to the site
Given the user is registered to the site
When the user visits the home page
Then the user is logged in to the site automatically
Given the user is registered to the site
When the user attempts to register with the same Google account
Then an error message is displayed informing the user that they are already registered
So the gherkin is not a problem either. It understands what the acceptance criteria are, how they tie into the higher user goal (story), and can format them pretty much however we like/want. This AI just keeps on impressing me.
Let’s try one last thing — let’s see how it fares when it needs to create a comprehensive set of user stories describing one large “epic”.
User stories (and epics)
Create a comprehensive set of user stories describing the process of a user achieving his or her ultimate goal of successfully selling their old bike using the online classifieds system such as craigslist.
1. As a user, I want to be able to create and post an advertisement for my old bike on the online classifieds system so that potential buyers can find it.
2. As a user, I want to be able to set the price of my bike in the advertisement so that potential buyers can have an idea of how much I am asking for it.
3. As a user, I want to be able to provide detailed information about my bike such as its age, condition, and any other relevant information so that potential buyers can make an informed decision.
4. As a user, I want to be able to respond to inquiries from potential buyers in a timely manner so that they can get the information they need to make a decision.
5. As a user, I want to be able to set up a meeting with potential buyers so that I can show them the bike in person and negotiate a price.
6. As a user, I want to be able to securely accept payment from potential buyers so that I can get paid for the bike.
7. As a user, I want to be able to ensure that the bike is safely delivered to the buyer so that they can start using it.
Ok. I’ll stop now. This is fairly incredible. It understands the concepts which are key to techniques such as story mapping, and that’s something that pretty much any intern or a junior will struggle with at the beginning at least.
So it’s obvious I’m impressed — and hopefully, you are as well (or at least intrigued), but you might also be starting to ponder things such as: Are we all about to lose our jobs as business analysts and be replaced by AIs now? Or if you are of a different persuasion, “How can I use this awesome new tool to make my job easier and its results better?”
We’ll discuss both of those questions in our next chapter.
Are we going to lose our jobs or gain a tool?
So the first thing first. Will AI models such as GPT-3 and their publicly available implementations replace us in the role of business analysts and functional designers?
The answer is reassuring “No.”
Followed by a cop-out of a “for the immediate future”
The thing is, that although the results of generational AI models such as GPT3 (or Dall-E 2 for that matter) are almost shockingly good, they are still somewhat “fuzzy” when it comes to small and particular details. And that fact is more due to the fundamental nature of learning systems and neural networks in particular than it is due to some “bug” that would be fixed in the next versions of the GPT-3 model.
Generally, it all boils down to context, or “what am I using these results for?”.
If you are generating a cool-looking title image for your blog post, small mistakes or inconsistencies are not an issue, and you can freely copy and paste that image into your post and be done with it. However, if you’re designing and documenting the functional scope of a system whose implementation will cost tens or hundreds of thousands of dollars (or your local equivalent), then those (mostly) small mistakes and inconsistencies can become very expensive problems later on.
So, you can rest assured, that until the models start producing *provably* correct, complete, and detailed requirements, we won’t be losing our jobs — and that won’t happen anytime soon.
But that’s only one-half of the questions we asked before, the other half deals with the ways GPT-3 can help us, rather than replace us.
Think of this example, imagine you are assigned an intern or a junior, someone who is very diligent, manages the basic concepts of requirements, is quite an erudite, and doesn’t mind producing a set of use cases, user stories, acceptance criteria or a summary for any given topic on very short notice — and, most importantly — can do it in a matter of seconds.
That is pretty much what GPT-3 can currently do for you. And that is amazing and in itself significant and useful. However, the point I’m trying to make is that any results provided by a junior/intern, are never such that you can just paste them into your SRS document/story card/whatever form of requirements documentation you use — without reviewing them and ironing out small issues that will invariably be present in such content.
Think of a restaurant kitchen situation where the chef “finishes” any dish which was mostly prepared by a lower skilled cooks — the reason that he or she is a chef is precise that they can take a “standard” dish prepared by someone with good grasps of basic techniques, and elevate it to the level where it becomes unique and special.
What I claim here is that our BA role in the context of AI-powered assistants is generally similar to those of restaurant Chefs and will tend to be more so as the AI models evolve and become better and better. And as a consequence, it will allow us to spend more time on adding that “extra value” to our designs and requirements, rather than spend that time on grinding a bunch of gherkin statements.
The “usage instructions” for our new AI-powered assistant
This part is still something that I’m trying to figure out myself in more detail, as this is quite a new tool in my “BA toolbox”, however, these two use cases are both obvious and useful:
- Creating an excellent starting point for any topic research (a.k.a. “googling”) — the ability to quickly create a summary (and more importantly — illustrate it with examples) for any topic is extremely helpful when you encounter a new domain that you need to design and document the functional scope for. Any summary produced by the GPT-3 AI model will invariably contain both the context and most important keywords that you’ll be able to research further using your “classic” research methods. And in some cases will be enough in itself to give you the most basic understanding of the domain and will enable you to further that understanding by asking it additional, informed questions. It’s the AI equivalent of googling and asking a colleague, and it’s powerful.
- Creating a basic set of documented requirements for a feature or a set of features, upon which you can then impose your hard-earned expertise and “Chef them up” to perfection
All in all, those two “techniques” however obvious and simple they might be allow you to both save time and get an additional “opinion” or knowledge input into your process without the hassle of having to manage the human interns and juniors who inconveniently tend to ask for luxuries such as pay, on-job education, and perfect working environment.
AI is very ambivalent to those, although GPT3 chat isn’t fully free to use, it is almost “zero maintenance” when compared to its average human counterpart.
The conclusion
Hopefully, this post demonstrated the fact that the AI models such as GPT-3 evolved to a point at which what they can do today was in realms of science fiction just a few years ago.
However, in my opinion, that’s NOT what is most intriguing and impactful about GPT-3 (and DALL-E 2 for that matter).
What IS their “life-changing” characteristic as far as I’m concerned, is their accessibility. Think about it, right now, without any hassle, you have (almost) free access to the technology that would be considered magic 20 years ago and the bleeding edge of the research just a few years ago. Also, you don’t have to learn some complex UI interface or communication framework to use it and it can perform some (menial) parts of your job — on a level that the intern/junior BA could be expected to.
It’s not often that such a powerful thing just “lands in our lap” for free. It seems that, if nothing else, it’s simply impolite not to make every effort to utilize its full potential.
And if you’re still pondering the question “What about me, the human business analyst? Will this thing replace me? It seems like a magical black box to me”, don’t worry about it, because, with all said and done, AI models such as GPT-3 are still rather far (and will remain so for at least some time) from being able to become a “drop-in replacement” for us, the “human” business analysts. Sure, they can write passable summaries, use cases, acceptance criteria, and user stories, but they don’t really work without an expert’s touch and supervision, and that expert, luckily, will remain to be you (at least for the immediate future).
Or in other words, AIs are not your competitors, they are your tools.
But as with any tool, there will be those who understand it well and can squeeze the last ounce of “added value” from it and those who “just use it”. I won’t be stating anything controversial here by saying that the former group is typically valued (and consequently paid) more.
So, my message to you is this — Don’t worry about AI models, and whether they will make your job obsolete or not, learn them instead.
You’ll get a twofold benefit out of that effort — first, you’ll get into that first, higher-paid group of people we talked about two paragraphs ago.
And second, as a business analyst (and especially as a functional designer) you should be able to add AI models to your bag of standard solutions that you can competently use to design optimal, modern functional solutions you’ll be proud of (and your clients will be happy with). Don’t make the mistake of thinking you can “sit this one out”. You really can not.
AI models in general, and your AI-powered BA assistant in particular, are here, waiting for your input and not going anywhere.
Make the most of that fact.
