When encoding video for home playback instead of streaming the goal is to achieve the best compression and quality in the fastest amount of time. When streaming games file size does not matter as long as a you do not saturate your network bandwidth. While for home use compression is king. I wanted to test what is the best format and encoder to meet these conditions. No AVC (h.264) was included as most video is already encoded in this format to begin with (AVC is commonly stored in the .mp4 container). This test uses handbrake 1.6.0 to test AV1 vs HEVC (h.265) to see if the AV1 is a good future proofing format.

For this testing a scene from the original Ghost in The Shell 1080p Bluray(Specifically the 25th Anniversary Edition) that I obtained from personal copy using Make MKV. For comparison of video quality I used the structural similarity index (SSIM) where 1 is an identical image and 0 is an unrelated image to the reference image. SSIM scores where taken from 4 different parts of the scene which will be shown below.

Below are the settings that where used to test each encoder using Handbrake.

Encoder	Preset	Encode Type
x265	Ultrafast	Constant Quality
	Faster	Constant Quality
	Faster	Constant Quality + Tune:Animation
	Faster	Constant Quality + Tune:Grain
	Faster	Constant Quality + Tune:SSIM
	Medium	Constant Quality
	Slow	Constant Quality
	Placebo	Constant Quality

---

Encoder	Preset	Encode Type
Apple Video Toolbox	Quality	Constant Quality
	Speed	Constant Quality

---

Encoder	Preset	Encode Type
NVENC	Fastest	Constant Quality
	Medium	Constant Quality
	Slowest	Constant Quality

---

Encoder	Preset	Encode Type
AV1	12	Constant Quality
	5	Constant Quality
	2	Constant Quality

---

Below you will see charts containing the averaged encoder SSIM values. The closer the value is to 1 the closer the image is to an exact copy of the original video.

Preset	SSIM
Ultrafast	0.953
Faster	0.966
Faster + Tune:Animation	0.960
Faster + Tune:Grain	0.965
Faster + Tune:SSIM	0.966
Medium	0.967
Slow	0.970
Placebo	0.969

It is important to point out that moving the x265 preset up doesn’t cause massive changes in quality. Going from ultrafast to faster is quite a big jump in SSIM value but faster to medium has less range than adjusting the encoders tune. Notice that placebo does not beat Slow. People have this idea that increasing the preset increases the quality but it really just takes a longer time to search for the optimal settings which may not lead the a better quality.

---

Preset	SSIM
Quality	0.943
Speed	0.943

Apple’s encoder appears to be broken in handbrake do not use it or buy a mac to use it.

---

Preset	SSIM
Fastest	0.966
Medium	0.967
Slowest	0.967

For our scene not a big difference between the presets for home use just use Medium or Fastest.

---

Preset	SSIM
12	0.960
5	0.969
2	0.971

AV1 with its 2 settings showed the the best quality of the entire test. It however took forever as you will see the in the test below.

---

Preset	Avg. Encode Fps	Percentage of Original (%)
Ultrafast	174.4	12.0
Faster	55.2	19.8
Faster + Tune:Animation	52.6	13.1
Faster + Tune:Grain	41.4	34.4
Faster + Tune:SSIM	56.0	18.4
Medium	29.7	22.0
Slow	13.8	26.4
Placebo	1.3	28.4

As you see adjusting the preset has a massive impact on encoding rate. Going from faster to medium does not drastically improve the encoding quality or compression but will take twice as long to encode.

---

Preset	Avg. Encode Fps	Percentage of Original (%)
Quality	114.8	3.1
Speed	114.9	3.1

Once again do not use handbrake to encode using the M1 macs built in GPU. The files look bad probably because they have to low of a bitrate hence the High compression ,but I used constant quality encodes so this is annoying.

---

Preset	Avg. Encode Fps	Percentage of Original (%)
Fastest	304.1	31.4
Medium	214.2	31.4
Slowest	116.8	31.4

NVENC’s setting just adjust quality but do change compression. NVENC has to worst compression of the whole test. While the encoder is fast it does not provide much compression. In some extreme cases I have found that NVENC does not provide in compression at all. I generally avoid NVENC since my CPU is strong enough to encode but you should consider it if you have a weaker CPU. I would not recommend buying a new GPU for its encoder best to upgrade your CPU.

---

Preset	Avg. Encode Fps	Percentage of Original (%)
12	261.9	22.0
5	6.5	21.2
2	0.4	24.2

The compression was middling but the encode times where some of the slowest in this whole test. Preset 5 had quality similar to x265 Slow ,but was half the speed and no where near faster which with tunes is not far off from Slowest. I would recommend AV1 as player support is poor and does not provide any benefits over the much more wildly supported H265 video codec.

Best performance

I created a top five list of best performing encoder. First the encoder setting had to perform above average and then second it must have the highest score consisting of the sum of milliSSIM per second and Megabytes saved per second. This tells us how much quality does the encode gain per second of encoding and how much file size reduction is achieved during the encode duration.

1. x265 faster Constant Quality using the SSIM tune (Score: 13.17)
2. x265 faster Constant Quality no tune (Score: 12.84)
3. x265 faster Constant Quality using the Grain tune (Score: 8.67)
4. x265 Medium Constant Quality no tune (Score: 6.80)
5. AV1 (SVT-AV1) 5 Constant Quality (Score: 1.50)

---

Below is a side by side comparison of our original with the winning codec. I want to say since our metric is in fact SSIM using the SSIM tune is basically cheating however I generally recommend using faster as it gives you the best encode time and image quality.

I am not going to show the others as the SSIM scores are relatively similar and the quality is very good for all of them. Below I will show Apple Media Toolbox which had the worst quality to give an idea of the range.

Apple Video Toolbox

Conclusion

I was shocked at how poor AV1 did considering the hype around it. It was able to get the highest encode quality. However, its’ compression was generally around than of h.265 and encode time was much longer. Remember H265 came out in 2013 and AV1 in 2019. AV1 is not even competing with H265 but the new H266(VVC). The real big push for AV1 is that big companies do not have to pay any licensing fees to use the codec. However, the total license fees are capped for HEVC (H265). Also, the big push for that AV1 is better from an open source perspective is nonsense since both handbrake and x265 are open source and open souce encoders exist for h.266.

Notice that increasing the “quality setting” does not lead to drastic increases in SSIM score. This is due to the fact that the “Higher Quality” presets spend more time searching for more optimal ways to store the content which may or may not lead to better quality. I would suggest sticking with faster for x265 if your doing constant quality encode. Also, NVENC had the worse compression and middling quality. If you want to encode video for compression I would suggest a CPU upgrade before GPU. However, if you cannot get near real time on the x265 Ultrafast preset then you should use NVENC as it will give you better quality at a much faster speed. Apple’s video toolbox handbrake integration is bad do not use it. If you have to encode videos on a mac use Apple’s Compressor not Handbrake.

Raw Results

Here is a spreadsheet with my raw results below.

Computers

• MacBook Pro (13″,M1,2020) 16gb of ram and 512GB running MacOS Monterey
• Custom Built Desktop with a 3950x

Processor	AMD 3950x with additional 200Mhz allowed Boosted & Infinity Fabric running at 1867Mhz
OS	Debian 11 Bullseye
Graphics Card	Gigabyte GTX 1070 G1
Ram	Patriot Viper Gaming PC4-35200 4400MHz running at 3533Mhz
SSD	WD – BLACK SN750 (WDBRPG0010BNC-WRSN)
Motherboard	Asrock X570 Creator
Cooler	Ice Gaint ProSiphon Elite

Benchmarks

To compare how your computer would run these task I attaching a few benchmarks

Geekbench 5

Computer	Score
Custom Desktop
-Single Core	1418
-Multi-core	16677
M1 Macbook Pro
-Single Core	1063
-Multi-core	4303

Phoronix Test Suite 10.8.4

Test	Result FPS
SVT-AV1 Preset 4 Bosphorus 4K	3.229
SVT-AV1 Preset 4 Bosphorus 1080p	7.649
x265 Bosphorus 4K	23.36
x265 Bosphorus 1080p	43.87

Phoronix test suite is a free and open source cross platform application that can be used to benchmark computers.