The best aspect of being part of the Innovation Team is working with cutting edge technology and, of course, “playing” with it! A few months ago, during a conversation with one of our clients, he proposed a certain challenge: “If you can do it, I will definitely take a look at it. However, I doubt that it can be done!” The challenge was: How to create a feasible test environment to automate Video Quality tests, for Video Conferencing products. Well, challenge accepted; and as we say in Brazil, “A mission given is a mission accomplished.”
How to Start?
Solving this challenge meant identifying a variety of methods, technologies and processes capable of automating a quality assessment; then developing our own hypotheses to test and measure the results. As part of Daitan Group’s Innovation team, our job is to be the front line in the organization that studies unique challenges, relying on our expertise to find good solutions. Every project is a learning experience and we try to discover best practices that can be shared and applied by others. I also spoke with multiple Daitan engineers experienced in this area for practical insights that could help me narrow down an approach. With that mindset, I was able to start the investigation and tech validation with Innovation team colleagues.
If you would like to know how we got the results and conclusions, you can link to the full article where we explain the details. However, if you have no time to read the whole article, I have summarized the main conclusions below.
After trying a few methods and processes, we found a small set of solutions that could be used to automate video quality tests, and provide trustable and consistent results, which resolves the proposed challenge. Combining Objective method libraries using scripts and Open Source programs proved to be the very straight forward path to solving the challenge. The difficult part was integrating these methods with the existing testing frameworks, covering the video quality tests demands, within the appropriate time-frame (as part of Continuous Integration and Continuous Delivery processes).
Therefore, it is our conclusion that it is possible to create a software program that gives a good estimation of the video quality, by using the appropriate algorithm for each scenario. Our results:
- Objective methodology for video quality tests can provide the confidence we need.
- Open Source libraries should be used to speed up the implementation and to assure qualified results (usage of commonly used algorithms).
- Almost all video tests can be addressed using different combinations of methods and algorithms (especially QR Code + SSIM).
Based on our results, we’ve compiled a summary table below with some references to the algorithms mentioned above.
Our Studies Covered
We have compared QR Codes with barcodes and used them to identify frames. We also created a method to generate, send and compare reference video files; and used those methods to check video quality, screen layouts, and facial recognition. And, there is room for more complex test scenarios.
Our Next Step Will Take a Look at Audio
As a next step, we plan to sync-up video tests with audio tests. By doing that, we will work on another common problem during video conference calls: Mismatch between what you are seeing on the screen and what you are hearing the speaker say. As we did with this project, we will publish our results once completed.
You can read the full article, How to Feasibly Automate Video Quality Testing, for details on the project. If you are facing challenges related to video quality testing methods and need help, please, feel free to contact Daitan. We would be happy to further discuss our experiences and share our knowledge on the subject.