bitfaster / base64

Extension methods that convert to and from Base64 strings with fewer memory allocations.

Base64

Extension methods to enable conversion to and from Base64 strings with reduced memory allocation. Strings can be converted in place, or written/read from streams.

Intended for use in legacy .NET framework (specifically 4.6).

Conversion via string extension methods

"foo".ToUtf8Base64String().FromUtf8Base64String() == "foo"

which is equivalent to

var b = Encoding.UTF8.GetBytes("foo");
var s = Convert.ToBase64String(b);
var r = Convert.FromBase64String(s);
"foo" == Encoding.UTF8.GetString(r);

Internally, these extension methods use RecyclableMemoryStream and pooled buffers to avoid all intermediate byte[] and char[] allocations. For short strings, it falls back to Convert APIs which are faster and optimized for the small case.

Stream equivalents

Since stream memory can be recycled, by keeping data in streams and not materializing string instances (e.g. write the stream to blob without ever converting to a string), we can effectively eliminate allocs.

Read directly from a stream as Base64 string (data can be written to the stream via any stream APIs). Thus, client code can avoid intermediate byte[] allocs:

using (var s = new MemoryStream())
{
   s.WriteUtf8AndSetStart("foo"); // write some data to the stream
   string base64Str = s.ReadToBase64();
}

which is equivalent to

byte[] byteArray = Encoding.UTF8.GetBytes("foo");
string base64Str = Convert.ToBase64String(byteArray);

The reverse is also possible, WriteFromBase64 will write decoded bits directly to a stream, stripping the base64 encoding:

string someBase64EncodedString = "Zm9v";

using (var s = new MemoryStream())
{
   s.WriteFromBase64(someBase64EncodedString);
   byte[] decodedBytes = s.ToArray();
}

which is equivalent to

byte[] decodedBytes = Convert.FromBase64String(someBase64EncodedString);

TODO:

• Fully optimized FromUtf8Base64String, based on char* end to end. Profiler shows that 20% time spent on ASCII string => bytes, 20% on bytes => ASCII char. Can just convert to char* from string instantly, save 40%.

References

• https://tools.ietf.org/html/rfc4648
• https://commons.apache.org/proper/commons-codec/xref-test/org/apache/commons/codec/binary/Base64Test.html

Using SIMD instructions

• https://github.com/aklomp/base64
• https://devblogs.microsoft.com/dotnet/the-jit-finally-proposed-jit-and-simd-are-getting-married/
• https://docs.microsoft.com/en-us/dotnet/api/system.runtime.intrinsics.x86?view=netcore-3.1

System.Numerics doesn't support bit shift operations on Vector. Need to be on .NET Core with the intrinsics APIs to try this.

Benches

StreamToBase64

Method	input	Mean	Error	StdDev	Median	Ratio	Gen 0	Allocated
ConvertTo	256	950.0 ns	17.72 ns	16.58 ns	944.7 ns	1.00	0.2384	1003 B
StreamTo	256	584.3 ns	11.70 ns	16.02 ns	578.5 ns	0.62	0.1717	722 B
ConvertTo	512	1,886.4 ns	37.55 ns	36.88 ns	1,865.8 ns	1.00	0.4616	1942 B
StreamTo	512	1,111.2 ns	14.75 ns	13.08 ns	1,106.7 ns	0.59	0.3338	1404 B
ConvertTo	1024	3,609.6 ns	36.22 ns	30.24 ns	3,606.8 ns	1.00	0.9117	3828 B
StreamTo	1024	2,237.3 ns	44.48 ns	75.52 ns	2,248.6 ns	0.61	0.6599	2778 B
ConvertTo	2048	7,036.0 ns	34.34 ns	32.12 ns	7,035.2 ns	1.00	1.8082	7595 B
StreamTo	2048	4,214.0 ns	46.12 ns	40.89 ns	4,209.8 ns	0.60	1.3123	5519 B
ConvertTo	4096	14,342.8 ns	282.10 ns	376.59 ns	14,131.3 ns	1.00	3.5858	15127 B
StreamTo	4096	8,209.6 ns	95.10 ns	84.30 ns	8,184.6 ns	0.57	2.6093	10984 B
ConvertTo	8192	28,121.2 ns	550.64 ns	565.46 ns	27,831.6 ns	1.00	7.1411	30144 B
StreamTo	8192	16,720.5 ns	329.76 ns	493.58 ns	16,554.1 ns	0.60	5.1880	21904 B
ConvertTo	16384	54,929.6 ns	369.26 ns	308.35 ns	54,981.0 ns	1.00	14.2822	60184 B
StreamTo	16384	33,705.6 ns	658.32 ns	583.58 ns	33,909.3 ns	0.61	10.3760	43752 B
ConvertTo	32768	117,409.5 ns	2,308.33 ns	2,834.84 ns	115,435.3 ns	1.00	27.7100	120232 B
StreamTo	32768	67,350.4 ns	853.04 ns	712.33 ns	67,137.2 ns	0.58	27.7100	87416 B

StreamFromBase64

Method	input	Mean	Ratio	Allocated
ConvertFrom	256	1.482 us	1.00	797 B
StreamFrom	256	3.474 us	2.36	264 B
ConvertFrom	512	2.796 us	1.00	1566 B
StreamFrom	512	5.296 us	1.89	264 B
ConvertFrom	1024	5.276 us	1.00	3105 B
StreamFrom	1024	8.522 us	1.61	265 B
ConvertFrom	2048	10.992 us	1.00	6183 B
StreamFrom	2048	15.547 us	1.46	265 B
ConvertFrom	4096	20.937 us	1.00	12340 B
StreamFrom	4096	31.543 us	1.53	413 B
ConvertFrom	8192	42.058 us	1.00	24628 B
StreamFrom	8192	63.658 us	1.52	561 B
ConvertFrom	16384	83.405 us	1.00	49204 B
StreamFrom	16384	124.832 us	1.50	1006 B
ConvertFrom	32768	163.363 us	1.00	98356 B
StreamFrom	32768	241.901 us	1.42	1748 B
ConvertFrom	65536	380.923 us	1.00	196648 B
StreamFrom	65536	503.032 us	1.32	3384 B
ConvertFrom	131072	772.906 us	1.00	393248 B
StreamFrom	131072	965.072 us	1.25	6496 B

StringToBase64

Method	data	Mean	Error	StdDev	Ratio	Gen 0	Allocated
ConvertTo	1024	3,697.6 ns	35.39 ns	33.10 ns	1.00	0.9079	3828 B
StreamTo	1024	4,104.2 ns	44.44 ns	34.70 ns	1.11	0.7858	3298 B
ConvertTo	131072	457,084.6 ns	6,405.96 ns	5,992.14 ns	1.00	142.5781	480656 B
StreamTo	131072	418,116.2 ns	8,025.71 ns	12,729.62 ns	0.92	99.6094	350615 B
ConvertTo	16384	60,829.2 ns	1,229.04 ns	3,585.17 ns	1.00	14.2822	60184 B
StreamTo	16384	55,634.1 ns	1,392.20 ns	4,061.12 ns	0.92	10.4980	44312 B
ConvertTo	2048	7,719.2 ns	152.65 ns	335.08 ns	1.00	1.8082	7595 B
StreamTo	2048	7,392.0 ns	145.82 ns	297.87 ns	0.96	1.4343	6034 B
ConvertTo	256	978.3 ns	19.58 ns	28.70 ns	1.00	0.2384	1003 B
StreamTo	256	1,917.8 ns	38.41 ns	74.01 ns	1.96	0.2956	1244 B
ConvertTo	32768	120,500.4 ns	2,284.94 ns	2,539.70 ns	1.00	27.7100	120232 B
StreamTo	32768	100,625.3 ns	1,979.01 ns	3,081.08 ns	0.84	27.7100	88099 B
ConvertTo	4096	15,362.9 ns	305.06 ns	759.70 ns	1.00	3.5706	15127 B
StreamTo	4096	15,423.5 ns	456.46 ns	1,345.89 ns	1.01	2.7466	11524 B
ConvertTo	512	1,973.4 ns	40.11 ns	117.65 ns	1.00	0.4616	1942 B
StreamTo	512	2,741.3 ns	54.74 ns	150.78 ns	1.39	0.4578	1926 B
ConvertTo	65536	238,210.8 ns	4,699.23 ns	6,432.36 ns	1.00	55.4199	240384 B
StreamTo	65536	193,640.5 ns	1,622.06 ns	1,437.92 ns	0.81	55.4199	175704 B
ConvertTo	8192	28,244.5 ns	110.96 ns	92.65 ns	1.00	7.1411	30144 B
StreamTo	8192	25,210.0 ns	419.09 ns	349.96 ns	0.89	5.3406	22430 B

StringFromBase64

Method	data	Mean	Error	StdDev	Median	Ratio	Gen 0	Allocated
ConvertFrom	256	1.431 us	0.0240 us	0.0225 us	1.422 us	1.00	0.1888	797 B
StreamFrom	256	1.434 us	0.0121 us	0.0095 us	1.437 us	1.00	0.1888	797 B
ConvertFrom	512	2.653 us	0.0411 us	0.0343 us	2.649 us	1.00	0.3700	1566 B
StreamFrom	512	2.750 us	0.0538 us	0.0736 us	2.762 us	1.04	0.3700	1566 B
ConvertFrom	1024	5.182 us	0.0357 us	0.0298 us	5.191 us	1.00	0.7401	3105 B
StreamFrom	1024	12.637 us	0.2507 us	0.3346 us	12.434 us	2.46	0.6104	2564 B
ConvertFrom	2048	10.099 us	0.0789 us	0.0616 us	10.093 us	1.00	1.4648	6183 B
StreamFrom	2048	21.218 us	0.4204 us	0.5004 us	21.106 us	2.09	1.0986	4619 B
ConvertFrom	4096	20.243 us	0.3419 us	0.2855 us	20.129 us	1.00	2.9297	12340 B
StreamFrom	4096	39.915 us	0.7939 us	1.1637 us	39.909 us	1.97	2.0752	8878 B
ConvertFrom	8192	40.005 us	0.6649 us	0.5552 us	39.743 us	1.00	5.8594	24628 B
StreamFrom	8192	73.411 us	0.6841 us	0.6399 us	73.551 us	1.83	4.0283	17218 B
ConvertFrom	16384	79.310 us	0.7240 us	0.6046 us	79.353 us	1.00	11.5967	49204 B
StreamFrom	16384	145.456 us	2.9057 us	2.9839 us	144.455 us	1.84	8.0566	34052 B
ConvertFrom	32768	157.254 us	0.9824 us	0.7670 us	157.071 us	1.00	23.1934	98356 B
StreamFrom	32768	282.910 us	3.5762 us	2.7921 us	282.367 us	1.80	15.6250	67566 B
ConvertFrom	65536	322.867 us	1.0735 us	0.8964 us	323.180 us	1.00	41.5039	196648 B
StreamFrom	65536	580.539 us	11.4230 us	14.0285 us	576.197 us	1.79	41.0156	134848 B
ConvertFrom	131072	727.617 us	6.1183 us	5.7231 us	727.427 us	1.00	116.2109	393248 B
StreamFrom	131072	1,161.373 us	20.9996 us	19.6431 us	1,165.704 us	1.60	80.0781	269048 B

This is comparing to crypto stream, memory allocs are excessive and perf tanks:

Method	data	Mean	Error	StdDev	Median	Ratio	Gen 0	Allocated
ConvertFrom	1024	5.212 us	0.0772 us	0.0685 us	5.192 us	1.00	0.7401	3105 B
StreamFrom	1024	14.102 us	0.2803 us	0.4526 us	13.966 us	2.73	0.6104	2564 B
CryptoStreamFrom	1024	340.905 us	6.6863 us	7.1542 us	337.914 us	65.36	15.1367	64173 B
ConvertFrom	131072	724.461 us	14.1122 us	16.2516 us	728.312 us	1.00	116.2109	393248 B
StreamFrom	131072	1,333.259 us	25.6557 us	34.2496 us	1,342.586 us	1.84	78.1250	269204 B
CryptoStreamFrom	131072	48,468.583 us	2,781.3085 us	8,113.2075 us	48,924.591 us	62.92	1937.5000	8138814 B