Setup a Video Streaming performance test from the AWS Cloud with UbikLoadPack & Redline13

In this blog, we will show how to run a distributed performance test from AWS using:

• JMeter + UbikLoadPack as the load injector software allowing to realistically simulate Video Streaming Players (HLS, DASH, SMOOTH, HDS)
• Redline13 as the web SAAS platform to:
  • easily deploy JMeter and UbikLoadPack
  • monitor your performance test during runs
  • get detailed JMeter HTML report after run including UbikLoadPack dedicated metrics and graphs
  • historize your runs
• Amazon AWS as the platform for load injectors

The JMeter setup

The JMeter Test plan is very simple:

We have a Thread Group where duration, threads, ramp-up and startup delay have been made variable, this will allow us to control them from the Web Interface of Redline13:

The Thread Group contains ULP Video Streaming Sampler which uses the stream and bitrate variables coming from CSVs and durationRead, conn_timeout and read_timeout defined in UDV_ENV (User Defined Variables):

And 2 CSV files to be able to vary Streams and Bitrates:

• profiles.csv (loaded by JMeter CSV DataSet CSV-Profiles) contains the bitrates we want to test, this element will fill in ${bitrate} variable on each iteration of CSV with a different value

• streams.csv (loaded by JMeter CSV DataSet CSV-streams) contains the streams URL suffixes we want to test, this element will fill in ${stream} variable on each iteration of CSV with a different value

AWS setup

AWS setup is very easy and described here.

Redline13 Setup

Note you need a pro subscription in order to have access to some of the features described below.

The configuration is pretty straight forward and is done in one page.

Click Start a New Test:

Select “Jmeter Test” Tab:

• In Upload JMX , select your JMeter jmx file
• Upload the UbikLoadPack license file you received by mail after requesting a trial at this page 
• Upload the 2 CSVs (profiles.csv and streams.csv)
• Select JMeter version (the most up to date one)

Configure “Advanced Cloud Options” options:

Ensure you fill:

• Location, in the below example we used Ireland and Virginia as a first step
• Number of servers
• Size (type of server)
• Subnet ID (just click in the field and it’s automatically filled)

In the below example we use 4 Machines:

• 2 M5.2XLARGE in Virginia
• 2 M5.2XLARGE in Ireland

Configure “Advanced JMeter Test” options

• In “JMeter Plugin support”, check “JMeter Plugin Manager” option
• Search for our plugin using “UbikLoadPack Video” test in “Plugins from JMeter Plugins”
• Click on the version of plugin you want to use, 7.1.7 at the time of this blog writing: 

• You should end up with this:

• Check plugin “JMeter Report” and fill in the APDEX thresholds you want (200 ms for Satisfied, 400ms for Tolerated in our case)

• In “Option String” , adjust JMeter properties as you would from command line. For the UbikLoadPack plugin you need to set those ones:
  • For license:

-JULP.videostreaming.licensepath=ubik-streaming-plugin.license

• • For fields saved and needed for the report:

-Jjmeter.save.saveservice.sent_bytes=true 
-Jjmeter.save.saveservice.connect_time=true 
-Jjmeter.save.saveservice.subresults=true 
-Jjmeter.save.saveservice.url=true 
-Jsample_variables=aws_region,bitrate,ulp_buffer_fill,ulp_lag_time,ulp_play_time,ulp_lag_ratio,ulp_lag_ratio_wo_bf,ulp_dwn_time

• • For configuration of test plan used for demo, we will set the properties used by test plan (notice the test plan uses JMeter __P function) 

-Jduration_read=60 -Jrampup=120 -Jduration=600 -Jthreads=1000

You end up with:

• • • In the above line , you may need to adjust parameters in bold depending on:
      • Number of Players you want to simulate per machine (threads)
      • Duration of your test (duration)
      • Rampup of load test (rampup)
      • Duration of video read in your test (duration_read property)
• In JVM_ARGS, you adjust JVM options , example:

  • -XX:+UseG1GC -XX:MaxGCPauseMillis=50  

  • -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false 

  • -XX:G1HeapRegionSize=32m 

  • -Xss256k -Xms6g -Xmx6g -XX:MaxMetaspaceSize=256m 

  • -Djava.awt.headless=true

  • In the above line , you may need to adjust parameters in bold depending on:

    • type of AWS instance and its memory (-Xms6g -Xmx6g)

Give a name to your test and click Start !

The test is now running, you can monitor it in the GUI and view while test is running:

• Request metrics (Requests per second, KB Per request, Avg response time)

• Error metrics(Errors per second, KB Per error, Error avg response time)

• Agent metrics (CPU usage, Network In/Out)

• The error table

Analyzing the test

Once your test is done, at the end of the test:

• you get an overview of your test in Summary
• you have the details of the run (JMeter and Cloud options)
• you can add Notes and comments on your test
• you can download jmeter logs from every server
• More importantly:
  • you can download the HTML JMeter report generated accross all server . Note that this report will contain starting from version 5.0 of JMeter the custom graphs for :
    • Statistics (response time in percentiles, error rate, network…) by type of request by type:
      • Manifest
      • Chunk
      • Subtitles
    • lag ratio graph
    • lag ratio without buffer fill graph
    • buffer fill graph
    • lag time graph
  • you can download the merged CSV accross all server that contains the custom metrics

About UbikLoadPack:

• Ubik Load Pack  is used by Big players in the Video streaming field
• We provide professional services for Load Testing
• Learn more about our streaming plugin
• Get a Free trial

You might want to read:

• Video Streaming Top 5 metrics to analyze your User’s Experience
• Load testing MPEG-DASH Video Streaming with Apache JMeter and UbikLoadPack
• Video Streaming: CMAF and Low-Latency

L’article Setup a Video Streaming performance test from the AWS Cloud with UbikLoadPack & Redline13 est apparu en premier sur Ubik Ingénierie.