6. SOAPSonar – Report View

An important reason for Automation, can be the time saved over generating manual reports. Comparing expected results with actual on a case by case basis and sorting through this date to combine it in a meaningful way. Making sense from pages for XML, in an attempt to filter the few key issues. One of the top 5 issues shared with me is false positives, or QA reporting an issue, that was either incorrectly diagnosed or not considered a issue. Much of this can be mistake in what was entered, but just as often, a mistake in understanding the expected behaviour.

So lets take a look at SOAPSonar’s report view, carrying on from the previous Tutorial #5 defining success criteria.

1. Lets start this time with JSON Google maps. In QA Mode, Project View. Please confirm you have the service and the [ADS] and that service works. Switch to Run-View and clear any tests under the DefaultGroup and drag over only the Google Maps test case we did in Tutorial 5.

2. look at the area to the right, In QA Mode there are 2 tabs. Suite Settings, and Group Settings. If you switch to Performance mode, there are 3 different tabs. In QA mode, lets change the Result File name to Tut6_1.xml and leave the location at C:\Program Files\Crosscheck Networks\SOAPSonar Enterprise 6\log\QA. We have not captured a baseline for regression testing, so select Test Case Success Criteria. Result Logging as Log All results and Verbose and Use Optimized XML logging. Warning, Logging all and verbose logs in large test environments can greatly effect performance and seriously load any workstation. We usually recommend log errors fails and errors. HP Quality Centre and other options, leave unselected.

3. On the Group Settings tab, lets leave things default. Here you can define a Data Source table to run through multiple tests. Commit and Run Suite.

4. Realtime Run Monitor shows that I ran 6 test cases of which 4 failed and 2 passed. Select Analyse Results in Report View at the top of the page.

5. In Report View, we can now see on the far left, under Today, I have Tut6_1xml Log file at the location we entered in step 2. Right Clicking on the file allows you to export Request and Response Data as Files and Export Results to Normalized XML. Select Export results in Normalized XML and you have details of each test case run.

6. In the main section you can see the first 2 tests are green and the rest red. Selecting the first test, and the tab Test Iteration Summary and the Request Response Tabs populate with the what was sent and received. Do you notice the exact time for and size of each message is also reported along with teh response code?

7. Select Success Criteria Evaluation, and you can see the Response time and exact Match rule we created, both were a success.

8. If select the 3rd line (first to fail) we see that it used the ADS Index 3rd row) the independent Response time and the response code was 200 (normally a pass). By selecting Success Criteria Evaluation tab, we can see that the exact match success criteria failed. That is, the csv value we were expecting was different from what we received.

9. We can generate a number of PDF reports via the drop down menu at the top of the page. These PDF reports can also be exported in a variety of formats. Take a look at a few.

10. Running the same test in performance mode and not QA mode, generates an alternate set of reports. The same is true if its is against a baseline.

These results can also be published to HP Quality Center.

Conclusion

Between management level reports and detailed level request and responses, reporting can consume a lot of time. Testers tend to focus on how long it takes to run a manual test vs automate the test cycle, often forgetting about the time taken to generate reports in a manual testing environment and how long it takes to capture and supply the required information with any issues. The ability to supply the test case and log, is key to troubleshooting issues, and the first thing we ask our customers for when they have any technical support questions. That is because automation enables the exact repeat of the same test case, and therefore any issue should be simple to replicate.

Comments?

5. SOAPSonar – Defining Success Criteria

Just because a response code is response code is 200 range or not in the 400 range, does not meant that the test met with with the business requirements a tester is required to test for. For a list of Status Codes look here. This is represented by the Success criteria or Baseline arrow vs Outcome.

Perhaps your test case requires a specific code, value, response time or some other form of validation. Responding with a fax number instead of phone number, or the wrong persons number, is still a defect. For this reason, SOAPSonar offers a variety configuration options, to define what is indeed a successful test case and what is not. Lets start again with our SOAP example we used in Tutorial 4, and use the same .csv data sources for calculate and maps

1. Launch SOAPSonar and Open the test case from Tutorial 4. If you did not do that tutorial, now is a good time. Check that you have both Automation data sources under Configuration, Data Sources by going and in checking the columns. Check also you have our SOAP calculate Service and JSON Google Maps service.

2. Lets use Subtract_1 service. Select it,. then in a= right-click and select [ADS] Automation Data Source, Quick Select, Calculate, Input a. Then in b= select [ADS],… Input b. Commit.

3. Lets run this in run view. Select Run View, delete any existing test cases and drag Subtract_1 under the Default Group. Commit and Run Suite. How many test cases did you run and how many passed? I had all 10 pass.

4. Now lets go back to Project View and define some additional success criteria. Select Subtract_1 and next to the Input Data tab, is a Success Criteria Tab. Select Success Criteria Tab and Add Criteria Rule.

5. Lets first add a rule for Response Time. Performance is after all a functional requirement and so should be part of functional testing. Lets just say 1 second as max value.

6. Now lets compare the result in column 4 of our .csv. Add Criteria Rule, Document, XPath Match. Then select your new rule and refresh if you need to. Look for SubtractResult parameter and right-click on it. Select Compare Element Value. Notice the Criteria Rules tab changes.

7. Select The Criteria Rule tab, then set match function to Exact Match. Then Choose Dynamic Criteria, Independent Column Variables, Calculate.csv, Subtract Result Column from our [ADS]. Ok Commit.

8. Switch to Run View and lets run this test again. Commit, Run Suite. This time 9 passed and 1 failed. If you check that .csv file, line 3 subtract value answer column is wrong. The result being 10, yet the expected value being 5. Without defining Success criteria, this would have being missed. Performance wise I had no failures and responses were good.

9. Now lets see if we can do this with JSON. Select Google Maps test, in Run View, clear the DefaultGroup, drag Google Maps over and Commit and Run Suite to make sure it is working. All 6 of mine passed.

10. Back in Project View. Select The Success Criteria Tab, and Add Criteria Rule, Timing, for a maximum response of 1 second. Then Add Criteria Rule, Document, XPath Match. Select it and look for the distance, value. Right-click on it and Compare Element Value.

11. Select the Criteria Rules, set Match Function to Exact Match and select icon Choose Dynamic Criteria, Independent Column Variables, your googlemaps.csv, Meters column. Ok, Commit.

12. Run Default Suite. This time 2 of the 6 test cases failed for me, although the time was near, in both cases, it was because my csv had a different value. We will look into report view in our next tutorial.

Conclusion

Being able to mix performance, and various header and message requirements into a multiple rule set to define success criteria, allows for automation to reflect business requirements. This helps ensure that you are not just testing Status codes, or incorrect functionality, but the full response. Taking the time to define each test case with the right success criteria initially ensures that your baseline, performance and other systems tests are more accurate.

The arrow from the enablers to the data sources in the diagram at the top of the page, indicates the ability to use direct SQL or other calls to enablers to compare the values with those found in the response. Allowing success criteria to include validating the service is selecting the right value from the enabler.

Comments?

4 SOAPSonar – Using Automation Data Sources

In Tutorial 2 we got started with REST and SOAP service, using design view and doing a unit test. In Tutorial 3 we chained the response from one service to be a request for a second service, creating a mash-up test between REST and SOAP services to run in Run View. In this tutorial we going to show how to use an external source like a .csv file for automating a range of values. This is represented in the test cycle by variables and data sources.

Why use Automation Data Sources? Well lets say you need to test a range of users, or values. Take Canadian postal codes as an example. There are 845,990 Canadian postal codes. Yet when testing, you may not need to test all of them. However each province, has 1 or more letters that start the postal code for the region. All BC postal codes starting with V. There is 18 possible start letters, each mapped to a given area. If your percentage coverage requires validation of address by checking postal code to province and includes testing negative scenarios, you could be required to run the same test case through 20 or more unit tests. If you have read more detailed service plan costing, you can see that 20 unit tests through many test iterations can add significantly to testing costs. The challenge however is to do this, without spending too long on the test creation or maintenance. If each release requires new scripting, the value of automation greatly decreases.

I am going to first use the SOAP calculate service, but show a JSON service example afterwards.

1. Lets first create our data source. There are many places on the web to find already created data sources, but in this case, I will use MS Excel to create one and save it as a CSV. I created 6 columns, input A, Input B and then the expected results for each service. I have only 10 lines, but you can make these much longer. Here is the calculate csv file for you to download.

2. Run SOAPSonar (with Admin Rights) and in the capture WSDL bar paste http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl. and press enter. Select Configure Project Data Sources. Here we define any Automation Data Sources we may use.

3. Now select Add Automation Data Source. You can see that ODBC, SQL, Oracle, Excel or File bases data sources are supported. Select File Data Source for .csv.

4. Give your Data source a Alias, Locate the .csv file you downloaded and ensure its still Iteration, and Data Series in rows. On the right of Data Variables field (were my curser is), select the option to refresh data and view data. Do you now see your .csv file? Select OK.

5. Now lets use that Data Source on the Add_1 function. Make sure you are in Project View, and QA mode and Select Add_1. Right-click in a= field and select [ADS] Automation Data Source, Quick Select, Your Alias (mine is calculate), Input A column. do the same in b= field selecting Input B. Commit the test settings.

6. Now select Run View, and Drag Add_1, under the DefaultGroup. Commit. How many test case did you expect run? Run Suite to execute the test suite.

7. You can see I ran through the entire csv file rows, or 10 test cases. You can Analyse These Results in Report View, but we will leave that and Success Criteria for another tutorial.

8.Lets try this with JSON. File, New Test Group. File, New JSON Test Case and paste http://maps.googleapis.com/maps/api/directions/json?origin=Toronto&destination=Montreal&sensor=false into the URI and make sure method is GET.

9. Lets add our data source Add this googlemaps .csv file to you Data Sources as a file data source. Remember to refresh and check it before saying OK.

10. Now in the URI highlight (delete) toronto and right-click (insert) and select [ADS] Automation Data Source, Quick Select, Cities, Origin. Do the same for montreal. Commit.

11. Drag your Test case over to Run View and commit and run suite. How many test cases did you run now?

Conclusion

You have just used a Automation Data Source in both a JSON and SOAP Service. Running multiple values through a unit test, vs manually entering these same values. What is important to consider here is putting the right values into the CSV file to test as many variables as needed. Using a Tool like SOAPSonar with a Automation Data Source, increases your coverage, and reduces the time taken to run test cases. Whats more, since it is not script bases, there should be little issue if any, running it again on the next code release., further reducing testing time.

Comments?

3. SOAPSonar – Run View with Chaining

If you have not, please do Starting with SOAP and REST Tutorial first.

Automation lets you test the business flow scenarios. So now lets model a very simply business flow. Lets take an example, the company uses a 3rd party API for calculating distance between cities. This is a REST service. They have a second API that they use to calculate. The Client application makes use of both. The test requirements are that users are expected to calculate the time it would take to ride a bicycle between Toronto and Montreal if you travelled and average speed of 20 km/h. To do this we set up out test cases in project view and run it in Run View. But to automate this, we need the response from one service to be the input into the second. This we at ST3PP refer to chaining services for automation.

TUTORIAL

1. Open SOAPSonar, and lets first add the REST distance service. In this case I am going to use Google Maps. File, New, Test Group. Then right-click on Tests in the Project Tree and select New REST Test. It’s always good to give names to things, so right click on New Test_1 and Name it Distance.

2. Paste the following in to URI field.

http://maps.googleapis.com/maps/api/directions/json?origin=Toronto&destination=Montreal&sensor=false&avoid=highways&mode=bicycling Make sure Method is GET and then Commit and Send.

3. In the Response section we interested in the distance. Unless Montreal and Toronto are floating apart, you should get 621 km at 1 day 8 hours. But at what speed? Select the tab Runtime Variables, then scroll down till you get the distance value field of 621476 (meters). Select this value number and right-click, Add Variable Reference. Name it Distance. OK, Commit.

4. In Tutorial 2 we used a SOAP calculate function, lets add that. Since SOAP comes with WSDL, just paste

http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl into the capture WSDL bar.

5. Now on Divide_1, right-click and Clone, twice. Renaming the the first to KM and the second to Hours.

6. Select KM and we need to enter the response from distance in Meters, into a = and b= should be 1000, the response needs to go into hours. in a= right-click, [RV] Runtime Variable, New Custom Group, distance. Set b= to 1000. Then Commit and Send.

7. In the Response tab, select Runtime Variables, then Divide Result and right-click. Add Variable Reference and name it Distance_KM. Notice the tab shows Runtime Variables (1) now?

8. Lastly we need to divide that response by 20 km/h. Select Hours in the project tree, in a= right-click, Select [RV] Runtime Variable, Calculator.asmx, Distance_KM variable. then b=20. This should be the result we have being asked to model.

9. Switch to Run View and drag Hours test case under the DefaultGroup. We should name that our scenario.Notice that Distance and KM are known prerequisites and so get added automatically.

10. Make sure we use the Test Case Success Criteria as we have not captured a baseline and lets Log Fail Only. Make sure you not Publishing this to HP Quality Center Project. Commit and Run suite.

Conclusion

You automatically placed into Real Run Monitor, but we talk about Report view and logs in next post. Save your project as Tut3.

You have just automated a test scenario of 3 unit tests across both REST and SOAP API’s that reside on different systems and at different locations (external) to your network. Yes you could have done this faster manually the first time, but we will look at ways to baseline and monitor as well as use this example for more locations or different speeds.

Comments? Please share any thoughts or provide feedback so others can learn.

2. SOAPSonar – Starting with SOAP and REST

So you have installed SOAPSonar and it’s registered as is detailed in this earlier tutorial. Were now do you start? First let me explain how SOAPSonar works. SOAPSonar is device independent, and requires no client or client application in order to test the Web Services API. It sends requests directly to the Web Services API, and receives the response.

The SOAPSonar Interface is divided at the top into 3 Views. Project View, Run View and Report View. In Project View we create the individual test cases, defining data source, variables and success criteria. In Run View we combine these test cases into automated test scenarios, by dragging or linking the independent test cases created.We can also define virtual agents, for load testing and logging option. In Report view we see the results of the independent test cases that are run, compared against the success criteria we established. The logs provide addition detail on for further trouble shooting errors. On the right is 4 modes to run the tests in.

QA Mode, Performance, Compliance and Vulnerability. SOAPSonar uses the same test cases created in Project View, and Test Scenarios in Run View, to run in any of these modes, yet provides different reports.

A SOAP Based API

SOAP Web Services API, provide a WSDL, which can be captured by SOAPSonar and used to automatically generate a UI. For example if you put http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx into your browser, you will see a free basic SOAP service example for calculation.

If thiis service seems to be down try http://www.w3schools.com/webservices/tempconvert.asmx?wsdl

1. Make sure you are in project view and QA mode. Place that same link into SOAPSonar’s Capture WSDL bar and add ?wsdl afterwards or paste http://www.html2xml.nl/Services/Calculator/Version1/Calculator.asmx?wsdl Then select Capture WSDL or press enter, you can automatically import the available services. If you look on the left, in the project Tree, you will now see a service called Calculator.asmx which has 4 services, Add, Subtract, Multiply and Divide.

2. If you select Documents you can view the WSDL and Embedded Schema.

3. Now lets see the automatic UI for the Add service. Select Add_1 service. In the main section, there is a Request section on top, and under that a Response section. Lets start simple and enter a= 2 and b= 2 then the green check mark to commit the test settings and the blue arrow to send current request to server. If you look at the Response section, you should see the result is 4.

4. If you select the XML tab at underneath the Request Section, you can view the XML Request that was sent. Changes in the Schema Fields or here are mirrored. Change the a field in the XML by changing the 2 to a 3 in <tns:a>3</tns:a> then then the green check mark to commit the test settings and the blue arrow to send current request to server. Does the response now read 5?

REST Based API

REST based API do not provide a WSDL that can be captured. Their lightweight nature, requiring a bit more knowledge or documentation.

1. To start a REST test, select File, New, Test Group. You should then get NewCustomGroup1 in the project Tree. If you right click in it you can rename it REST Demo.

2. If you Select Tests, then right click and and select New REST Test, you will see a similar but slightly different Request and Response section.

3. Lets Check the Bitcoin price. In the URI in the rest section, paste https://btc-e.com/api/2/btc_usd/ticker and leave the method as GET. then the green check mark to commit the test settings and the blue arrow to send current request to server. What did you for a response? My high I got was US$584 today.

4. Notice that by pasting in the URI, it is then broken down into Protocol, Host, and Path. Right click on the New Test_1 and rename it BTC_Bitcoin.

5. Please save your test plan at this point for the next Tutorial on Run View.

Conclusion

Well now that you have some idea how to start a REST or SOAP Test case development, take a few minutes to try a few additional service or services from your own environment. Try a BBC Music artist

http://www.bbc.co.uk/music/artists/ada7a83c-e3e1-40f1-93f9-3e73dbc9298a

or SOAP Soccer information service

http://www.xmlsoccer.com/FootballData.asmx?WSDL.

For that matter, Facebook, twitter and many other companies offer API’s that can be used, although some require a developer registration and key first. Perhaps you have a service you familiar with you would like to try?

We now have some idea of project view and how to start with a SOAPSonar project. next we will look at Run View and how to use some of these unit tests.

Comments and question below will help others learn.

1. SOAPSonar – Installing and Getting Started

So new you new to SOAPSonar. Perhaps you joined a new team that uses it, a student, or you downloaded a trial and trying to get started. Here is a quick tutorial on the install and UI.

Installation

After downloading the latest version (request a no obligation 14 day trial here), install by right click and run as administrator. If you don’t install with Administrator rights, you should get warnings.

Once installed you should be presented with the registration screen. Please enter your name, company and email. If you are behind a web proxy, please first enter the proxy settings. These can be taken from your browser settings. Go, File, Settings and Preferences and select the Global Proxy Settings tab. enter your organizations proxy settings here before registering. Alternately you can select the Manual Activate button Option.

If you doing and evaluation or your company uses Instance based licensing, enter your license key then select activate key. Instance based licensing can only be loaded on one machine per license key. Note if you using Instance based pricing, the number of days still on your license is shown.

If your company uses Floating licenses centralized license server is used. The server is used to check-out or check back the license only and does not require persistent network access. Licenses server is only accessed at the time the lease is granted or released. The server is not a file server and need not be a dedicated machine, but requires a static IP address and a small windows application. SOAPSonar can then be loaded onto as many machines as you would like, but only one machine can use a licensee at a time. If you using floating licenses, select the button for Use License Server and enter the details. (or hostname) and open port. Request New License Lease and select the Key Type and how long you wish to check the license out for. Select Request License and if there is an available Floating license of the Key Type requested, SOAPSonar will activate.

The User Interface

SOAPSonar UI follows the usual windows convention

File – File related tasks like new project, load project and save project with Settings and Preferences
Mode – SOAPSonar 4 pillars of QA (functional), Performance, Compliance, Vulnerability (security). Note the mode can also be changed in the right corner.
Tools – Various tools from key management to traffic capture.
Library – List of Automated Data Sources and vulnerability definitions that can be used.
Updates – To check for updates to the latest release. SOAPSonar does not require manual scripting. If manual scripts have not being used, upgrades should have no impact on test cases. It is highly recommended that you stay current to avoid known issues and get the new features.
Registration – Discussed above
Simulation – To launch a CLOUDPort generated run-time virtualized mock service to test against
Agents – For downloading and configuring remote performance load agents.
Help – Yes there is a help file.

Now each of the 4 pillars (QA, Performance, Compliance, Vulnerability) can be set to

Project View – For the configuration of your Project Tree or Test Suite Groups for automating your testing.
Run View – For running automated tests projects
Report View – For those who actually want to see the results after a test is run.

Conclusion

You should now be ready to start testing. If you have a project and feeling ready, you can start that.

Else we are working on developing Tutorials. These will be very basic tutorials to highlight a few frequently used features that should not take more than a couple hours to complete.

Tag Archives: SOAPSonar Tutorial

6. SOAPSonar – Report View

Conclusion

5. SOAPSonar – Defining Success Criteria

Conclusion

4 SOAPSonar – Using Automation Data Sources

Conclusion

3. SOAPSonar – Run View with Chaining

Conclusion

2. SOAPSonar – Starting with SOAP and REST

A SOAP Based API

REST Based API

Conclusion

1. SOAPSonar – Installing and Getting Started

Installation

The User Interface

Conclusion