400-8166108
行业动态
在这里,聆听大咖的声音
小白学大模型:训练大语言模型的深度指南
由 辰智信息 发布于2025-03-10

在当今人工智能飞速发展的时代,大型语言模型(LLMs)正以其强大的语言理解和生成能力,改变着我们的生活和工作方式。

在最近的一项研究中,科学家们为了深入了解如何高效地训练大型语言模型,进行了超过4000次的实验。这些实验动用了多达512个GPU(图形处理单元),它们协同工作,为模型训练提供了强大的计算支持。

在这项研究中,研究人员特别关注了两个关键指标:吞吐量(通过标记的大小来表示)和GPU利用率(通过标记的颜色来表示)。这两个指标都根据模型的大小进行了标准化处理,以便更直观地比较不同模型在不同硬件配置下的表现。

https://huggingface.co/spaces/nanotron/ultrascale-playbook

unsetunsetAI模型训练的三大挑战unsetunset

内存使用:有限的“容器”

想象一下,你有一个装满水的杯子,但你还需要往里面加更多的水。如果杯子满了,就再也装不下任何东西了。这就像训练AI模型时的内存问题。如果一个训练步骤所需的内存超出了GPU的容量,那么训练就无法继续。内存是一个硬性限制,我们必须在有限的空间内完成复杂的计算任务。

计算效率:让硬件“火力全开”

我们花了大价钱购买高性能的GPU,当然希望它们能时刻保持高效运转。但现实往往是,GPU在等待数据传输或者等待其他GPU完成工作时,会浪费大量时间。这就像是一个生产线上的工人,因为原材料没送到或者上一个环节还没完成,只能干等着。为了提高效率,我们需要减少这些等待时间,让硬件尽可能多地用于计算。

通信开销:减少“内耗”

在多GPU协同工作的场景中,不同GPU之间需要频繁地交换信息。但这种通信会占用大量的时间和资源,甚至会让GPU处于闲置状态。这就像是一个团队开会,如果沟通成本过高,那么真正干活的时间就会减少。因此,我们需要巧妙地利用节点内部(速度快)和节点之间(速度慢)的带宽,并尽可能让通信和计算同时进行,从而减少通信开销。

从单个GPU开始:AI模型训练的第一步unsetunset

在探索如何用数千个GPU训练大型AI模型之前,我们先从最基础的部分开始——在单个GPU上训练模型。这就好比在学会驾驶飞机之前,先要学会驾驶一辆汽车。别小看这一步,它是我们理解整个训练过程的关键。

bbbh5dmEJJdT5Vn0rE9i5Q+0/WLNrjkENjlnXj7VGHHZju2h48embBdNeO2m9n1VaxKu/PLOymJnyo5kBd1pGaJFxiAc4+Hm8fT3C4TnSUlmXrLbDjMgJA4cstm4P9XYYEFkrhKJsfFbS4bDGs/ZaFKPR0l9NN6RTiAMZ+C8p280Js4gnaxHkddmrJtmm5g3Xa2TI6GItnzCzNo4NYpx3M1CKiU8LMwriRT4nWH27MOVp7i5hUDLQtUxeoIASAwpdhQge7vNwIPJ7Cw8PDFbkivdNs48o3XpS5+flWGBmuZ5ozqKTB+1e+O/PN2883GNnW8V6HnVtzbFtzcU47W9FDMXhGFJoShabE4BmxBCZSWtBDeKcdrNQiorRvPf3kLvNGATVGH9WUc6wuhZRcArwFCEwjABS+ENmcxxey2Dw2l89Z3IXN5bM4PA73l5vgFwIFaHMxEFBU4QKBYGJiwsXFBYfDPfIBq0DhMlU/rDJN4c4l1e9+vvrTb7/zqm+bqfDIo7oNBr7TEjRQ+DQg4OM0AkDh855hBUKIw5W+rVhZ/rh8gexRNo+kQaMzSGQygUicWUhkCp3xy0PsH9kOWOBZElBU4aampiKR6PTp00lJSe2/f0sMEq78o12Awh9mbtn0mQrfoaGt7hM0ay8cKHyanMBHRQgAhc97JqUz2dNeRbj4Xc5kS2+bfmQZRo2kpqaiUKiB2f5GRqRzsdil/7jZR4JabAsoqnAikTg2NmZnZ6fIQHq/5Q+Hv3hL9cu3zDZ/iLVT9NI2md7mtyIbSG832Xh0Kqqoo2vmdxNP0NpMhZ90dLPNKwYKV0ROYBlFCACFz3u2JVEZi9/Z0yJksXmKcKhvaJy24rSPQqGws6tLkabAMs+SgKIKp9PpY2Njbm5uiij8CZS2cKvIFL5wm3iClmcq3PBGmmwIfdq58ICDmv/860tvvPh3lE2ULHeDgXQZClCZlQBQ+LxnUhKFPk1si/8ji6OQwsvLK+beF4FA0NjUPO9IQYNPSUBRhbu7u8MwDBT+BLaedZVpCp8m72kKnzVBA4XPigVMlBF4NgpvxkBR94Ue+YLoWmE77nevCX/K3LQIVwcKX4QHZZmHpKjCFX/AqvS+cMftBJ/jj12u7se7/ETwPf7Igr+6F+/yM9H72CMLCcEcCAAAIABJREFU4ep+WS8c77zrsUPyOY53+RnvtveRIRF8pUsS3A88MiTpAl6HgcJlpgGVBSKw0Apn8MR+xYKvHThf2/9aHDjhlUI2/9FnXpU07c6qcB5fIBJBsi4sny/g8fmTk5OyKc+3AnrhSvplUzDsBVE4KfAMGcUcbKY/VuHUZjNSnVsplEoCcY5SSyRRSuPpyXas1AJWyiMKr+6OTOHUOGN8/2NHRUtxoRZH15JIc4RUSSC2UCiMDDduTfojQ2KlFPBqaoDCF8hboFkZgYVWuEehYJ28v6dEvs6RE1C2ZG9kmqbwiYlJr4CIM5oGR9Uu9Q+iJiYm3H1C1bQMD5zQKK+qfb7mlm0dKFxBFyrpYguicHLwuZE+cW2R6LEKt6WYmeWRQ+XMOqosm5hEYhGqbtFvOpKcE2XZ6mEVQWupTOG0RPO+NvixQqotEtEzvEjlN5JJLFkAs1ayqRxmjg+vNvthkchP5xRUAYXLAwH1hSCwoApvxUC/63/LOuL2nK8dOO1oHouzdAqHy0desDZN4Qwmy9jaBYKgkvLqwPBYEpmqcdlcBEEEIlnzsplMos+3AhSupG5WMOyFVfj9QkF1AV9Ba8oUfg1NjhgmRI6QZpWlvMKx9nFz575ZFV6dz6spFCgYlUzhkSOkiGHCtVHyrFHJKxxjGzN3VEDhc/MBc+eFwIIqPLZW8Nv4uby/p+rXq2gYPHkpFQpNelf0NIUjbh7FYNW0DLLzCmWqHkFj9U3tZR+fbwUoXEEXKuliC6vw86esft6hqqAsZQr/Zp+KmoevcXJ6NI4+05fyCtf83865k92sCt+947TaSVMFo5IpfMPRE9ph0Ta5RTNDiiUwZQofsrq24cOVc0cFFD43HzB3XggsqMKDy+ZSeFjxUlM4lkB5mMJr6ptsnL0jY5MQVTc2tx04fqFvYOj5mlu2daBwJXWzgmEvoMILs0hrP//uP6u/z0zsV8SXMoV/vXvfbj0DzaCImAVQeG4q+uRhgw/eW1WURVIkKpnC1+3Zf9jC1iw1Gyh8XgQDGlloAguq8NTmuRR+s3ZJKRxLoNAZ7JkKhyAINYqRSCRi8dgJ9cuTk5ONLe0n1fUIJLLMoM+9oqDCq6qr5w5VIBC0tLYq6BWw2DMjsIAKtzAM2bvrrOpRAy11e0VkKVP4dwePuFc3hQ/hZpXlU/bCD+7TPHlY/+DeC6E+BYpEJVP4Li3d0H505Ahx1qhAL3yhhQTaf1wCC6pwNA3+2unXC9GnDaS7cDA0EV+wdIpA+MvNctMG0ukM5oETGi3tXYkpmSbWLhgsYfPu47fS71TXNja2tM9txGc2V0GFd3R2lZSUzBFVYWHhwODQMzMT2JCCBBZQ4Xbm0VX3uLVFIhebREVkKVO4ZmB42ABmVlPGEpjyCg9Q0bz8/Z7L3+/BPeSk+LSB9K4m/vnTVtX5/JIcWphvoSJRyRSuH5f0sJDkB9LRtjFGP6pc/n6P+ZZDBIfZX/oJBtIf10Zg+ScgsKAKh2BxRqvomxkWX+/MKeoTKZh9lG6xaQqXSCTVtQ1OHgGe/uEUKm14BO3pH46UiJgbc+jwWc5SUOEQLK6ursnOzkmZ7S/nzp26unqlO17LIWBFFT45Oclms+Pi4hR5Ohu4In3uhAsUPjcfMHdeCCy0wkWwOKlR9D9nDnJr2ToHzv+cORmtS9bfMwfSn6WJn3hbiit8OQhv6e2jogqnUqkwDA8MDACFz9oXlw2kK5J8gcIVoQSWeUoCC61wJBuiqHBpP3SnXVQ2AI3S4KWXIuX3aGYv/InN+sxWBAqXP4JLr66owq2srHg8XmBgIIlEqqyunQlC/k1loBc+d/IFCp+bD5g7LwSejcJnpoIlPAUofAkfXCXdNUUVTiAQxsbGXF1dCQRCSXnlzL0FCgePdpkX8YBG5osAUPjMNPWUU0hUxmJ5bqrCvXgF31T2lGTA6s+LgKIKJ5PJQOGzDqEjE8FA+nyJB7QzXwSAwuc9qzJYnPHxCYXtuSgWZCn2vvB5ZwUafDYEFFV4Xl6eWCwGvfCHWRwofL7EA9qZLwJA4fOeQ4UiiMXmLp5XmMz9G0F6DTKHz+ML550DaHDxEFBU4dXV1TAM9/T0cLncgeGRmTsgP5BOCjzDJnLJI49X2DVZjBQHNJs1wGTOUYaYTHppHP2mg6ChU1D/iMK7f1v2jHRqnCET/3ghkUe4tFvO9KKoYdZcIQ0wmaMsFiPNhVeb9ciQBPWdwvZm8Iz0+RIVaOdhBIDCZ6app58iEIqYbC6NwaYyWIu50BhsBovLFyzlGwSe/mgugRYUVbjiLxutIfaR/E4unsK6F3ymOYNBHVk8IUkjidR2bsu1bc3FOe1sRQ/N2rlvQQ/hnXawUotmzdHgfeGzYgETZQSAwpdAgga7AAjMTWD+FW7dXezfXbKISl/Fturr/oP3F1FI3SX+PaUnGtOAwmW+AZV5JwAUPnfuA3MBgSVAYP4V7tBX5tdbvnhK0ED1jvsJoaiGxROSX2+5f1+FalM6onBqXhC5OGZmoeYF4Ry2sXJDWbnhMws7P4rkr4pqyjlWl0JKLpl3AYAGlZ0AUPgSSNBgFwCBuQnMv8Kr8D04+y2Lp8jOhS+ekHD2W5Bz4Tbtd/Ep9vibNk9abPt6KoDCld21CxQ/UPjcuQ/MBQSWAIGFUviI7eZFokx5hY8umqgQhW+sjDlcm/w0Zc/9RKDwBVKgsjcLFL4EEjTYBUBgbgILovBOs40r33hxsSn8/pXvznzz9iKJClG4WXveSGP2SEPWk5bbtUMNQOHK7toFih8ofO7cB+YCAkuAAFD48xnz/+2mMuddOIetT1gct6Ga7gCFL5AClb1ZoPAlkKDBLgACcxNQVOFkMrm6ujo6OlooFOKJlJmNyt8XDnrhj+zr/6Zwp50Yuyc96eCwFVzOpuyiXbj4F07hjS0d+cW/PGU5+24RgUiJiE2qvN8gSwtBEfFdPQOyj34h0d19g7KPD6swWRw7V187V1+vgMiySumLGPoGUVHXbz1seQWnxySkVNQo9KJMBpNtbOVCoTFcvYK5PAHSPo3O9PALE4qW7Btc8osr7Fx9Ha76R8QkMdkcBak+2WKllbVkKn3aulQao6W9G4LF0ddv1dQ1TZsLPs5NQFGFt7W1HThwoKWlBTWCKiwum9movMK7zTd9+vpLX775t8Nr3sDYPZ9urkyisnPhjYYbVk5F5frzJ7K5z6sir3DjLR8+YRhA4Y4JC6dAZW954RSelVvgHxqDJAFX7+D+QRSJQmOxuUIRPILGDaHQZrZXG1s6hCIYhcYOjaANLZ2a27pEEIzG4geGRtgcnlAEjaCxOAKpf2hEJksShfbd9oPNbV33iiu27z9VVFbN4wvJFBqby8MTyUMoNBZPxBPJfYMoZBUyhdY3MIwogUiiEsnUvkEUjc6EYDGVxugdGMbgCCIIJlPpU7FBGBxxYGiEzmRBsBiDI2LxxKERNJfHl2WzzDsFxlbOECy+eMVyeBQjDZXLg2CxjbM3lkCEYDGTxRGKINnyS6MSEBaTV1DWNzB8WsMgNStPKIKweOLA8AiLw4VgMYvD7R9EjWLwIggWQTAWT+wdGKYxpAyxeBKBROntH6LSGYMo9AgaJ4JgDI6IJ0iPEZXGgGDxCAbH4fGnDjeOwWRv36eam18iFEEkCm3qABFFEJxfXGFg7kCi0MgUGovDFYogDI4wdaTYD44FgUQhkqmDw6NMlvTnBZPFEUFL9ufUE3yjHkPhcXFxY2Njt2/fzr1bMHNL8gp/QiEtzHXsMoUvqqiAwpVdkIs//gVVuIW9R31TW31Tm4GFU/8gysMvrKC06lbGHdXz+qFRCVv2nGhs6UhKzT6taRAanbh59/Hmtq6023mXTWwj45LtXHyoNMbPh886eQQYmDveSMlC8gmJQvt+x2ECSTrIl19c4XDVv38Q5R0YWVPXtH2/alBE/A8/HXX2CDSzdfMJujYwNHJUTSfq+i1NPXMeX2Dn6nv2opFXQIS+qb0Igi8ZWienSbfe3NblE3StoKSyqKz6or6lX0j0UTUdMpWuclLDwy/c0MLJO/CaLJu5eAYVlVVDsPiomo6nX/hlEzsTaxcIFt/KuBMVLx0MUL9k0jeIki2/NCoBYTH365tFEOzsEZh4K6uiul7P1M4vJPqE+mU6k61tYBUYHqtrbJtzr/heUfnJ83oRsUkaumYCIXTJ0FrP1N7G2fvQqYsRsUkqJzXbu/rULhpqXDYPCIs9cuYSmUrfc+RcU2sniULbffjswNDId9tUvAIisHjSqQv64TE3dqmc6ezp9w+NOXler76pzSsgsqTifn5xxcUrlr7BUSfUL1NoDGMrFysHTxtn7/M6piIIPqdtstBDBcp1WB9D4YmJiWNjY01NTUDhT/9rACh88StQ2SNcUIUfOXPJ2snL2snrwEnN/kHUVd/Qu0XlZ7WNkQFwfTOH+qY21Qv6lVMj2DpGNo3NHUfULlk7e0dfv7Vt30k8kbzr4Bk8gUyhMUysXZGkKa/wzp4BU1u3voFhT/+I6tompHN8Xseks6e/sbnjtIZ+1PVb+49rxCSkXNA1q2tstXXxySsohWCxpp65CIL1TOx9Q6JLK2u5PIFXQER+cYWNszdi33PaRo0tHac09ClUelNr57Z9qgLhLx1rM7urvQPDECw+r2PK5vCa27r2HTvPEwgLSirtXX0hWIwnUfiCpfbI8YCwGE09C/VLJjtVTo+gcbYuPh7+4dEJKet+3JdXUHryvB6XJ6BQ6f1DI/pmDgUl0hMo1k5e7d19F69YVdTU8/gCbQNrCBZrG1rfyS85paGfV1AqFMHql0yKy2t2HTzT2NpBotB2HTwDweJdKmcaWzr4AmFlTUNiStaOA6fyCspa2rtsnLwgWOzhF15cXmPl6Dk8gnnQgz+nbdzS3m1o6YQnkDE44pa9Jyg0Bp5IAb1w+R8Zj61w/v9n7z3Aokryvf/n+cf3/u+z93333t27u/fu3Rzm7r47M+qMWcwBJKhkkJwl55yjICCI5KRIFBVUJCkiOYPknDrn3H26SX/xzJzp6SY0SGPTFE89M3Xq1Kn61fe0v8+pOnWq2OzpWZRwEXAc9MK3xHVhhB/73f/66f/zfxz9zf/cUgmrmcFAOhhIX18BqSJcZCA9/E7Sq9p6I2uX6tcNEF9g4+zb3tWnZ+bw+m3Tx3Fpr86efg0D68aWDhaby2JzCSTKRTV9EplKYzBdfcJEEM6F+G6+YfH3sxCE+4ZEfyCouZ3H4Mh4Z0+/nqldfklZaHQiXBoX4vuGxMBPD7aufhBfwOVBE1Ozajpmj59VwAgPDI8bGpmA+AIto1vdfYMGFk5EMrV/aPQjwr9bSHyVTAPDsMEMJmtwZFxJw5jF5lRU10XcSRJ3evKREn8/s7Glo6t34MBJJQKRHBqd2D84uioshzs+OatpaMNgsnEE0uDwuINH0PPK1x/k9fCPHB2ftrD3amnv4fIgWHM7V//nr1YRXvay5oP++uaObxvblNSNGlo60FjCOWXd7xDe9X5wZNzQ0hlHILn6hNbUNXb3DXgFRCEI9wu9Mz4582EmhLbRrb7+YTffsNUxdiL5tKKW+Ht0+bgFn9IKSREu+RrpDeghbPQNRksZ9d2TLQXK02hihh259Smx+fFGoaWUVBxESLdlvH7EqN0kUMvuIAPp+CQTevOWrSKk25IK/YktpRuZ1Px41ewsB2r5nU1N+pjhgTDCt0xu5I0DQPj6ANvrfehPt196CH/2vFoc4VWv3zW1dmoYWFs7+V7VMO7s6W9o7tA0tLF28lW8YdTdN9Dc1m1s7ersHRoanSiMcDff7xCOJ5IPnFQyveVuYefpFxpDIJJHxqei4ld74TDCLb5H+E1zBzQWb+ng7ewVYu3oTabQhBHO+/gE4OYbrm/mMDA0Fn03tbK2vrtv0PSWm42zn5tfOJXOWBPhCcnZDwueiCGcm5qdX1lbD/EFti5+MF0+xefK2rXxSVnNbV1cHuTqE5adV9I3MGzl4G3nFuDoEUSi0KLiVl8oGFu7dvUO9L4f0jO1t3X19w2J4XB5Qgj3//DW3NEj6EXlG11TOws7T2snHwf3ICqNUfzkpaahjaNn8JFzahBfcC8tV9/ccXxyRv2mlYd/xHU9y9q6xtl5tIqWSdGTF1Fxq73wzp5+ExtXG2dfz4BIGp0pgnBbF1+wfarwT0gqCMcnGk6PCFpqeFsKzO5a6tPIciJjzT0/kMRHOBqmoZBcEIgLfripm+P0vEEQTnroPtLL35JJLTU8cult3Nu8fBwNMWDNyKdsNgoQvul9BBm2oYD0EC7sQUTiHC7E4f6wOxaHyxOe/8XlQTv7IpNKZ643rEpnsMRPCU9eE7Ec4gs6e/rVdMxYPx4q53B5tq7+8AQ64baIXy5PKYyP8/jgFjE/zkeD4zyIj0w/XLO9uqZ2Dc3t8FQ4OAODyRa+EbCGH8v5YSIhPFdOuMD1atk/t0BYjQ3i0kV4czW3qYotITURhKfPEVKncWkz+DVhKYxwlH/2xq5tTYQ3VrKaq7kSWoUgPG0WnzqNS58lrGmVMMLn/bI2topR1QB64RtLBM5+ugKfBeEb+BrZP8WD+HXvWijU1YnQSCCSqU2tXcghiGysQG1dIwqD2zgPOLuDCkgX4TamoddVLCWEJYLwA1eunje11Au7nTFPFOelMMLNDl/Y2NOtiXClSwa3zMMktApB+IErV5XsnW3ScsRNysJQEYRPeKUd/e1fNrZKBOHUsQnUGHOrgYRiETPtwHfhG0u9n88ChO+gowRFAQVkUwEpIvxdBf1vfz3y1z9/86pkXhJe/oDwy0rnzSyNYu5mosjivPxEhNe9oOhpuPzqF3+oe06RxCoE4V9fVlRycHHOfyxu0icifLZrRhJLRPKMdAKEg4/CN1IAIFw2fS6wCiiwgwpIEeGBXjnnFdRvqFg6WN8Wwc+ahwjCD6te93/1Jq53NAtNEeflJyLcQMddQ9X6ynndtLtv1zRDJBFB+Dlj89ttfYnD0+Im7QjC0xPqXe3iRWrf4BAgfD/3sCVpO0D4DjpKUBRQQDYVkCLCzQx831XQW2p4Eo5aIwhXc/FMHJpak5RZGKowwp1OqSr87m8Kv/vbeq+fRQbS+zuYJ49dbaxkVj/Fhfo+2gCQyCkE4frh0euZJIzwaZ/0S389oPC7v6n89zfogJw1Xa3IQPps10xTFefMKTW1q2aSv6QHCF9TW5CIKAAQLiWfy+FBLM7qp3EyHpDv3SXRYa80SsY1l5Z5HC5nnWUBJUX4wsJCTEwMkUiEIAhLEF3k9sNiCMLfhYMZ6YgbXTMijvAXRTMKx5Q1r9mmJ9QjTw8bRwDC19QWJCIKAIRLgq6t5qExWHTG6lRqwcKCLAe+QMBkcai01UVSNw2rjWLugUbJsuBStQ3iC+gMNp2xutyvSJAU4YuLi8+ePaPT6QDha/bFkelsiAPdICKO8KigEg/HexEBRV/85aCEc/gBwjdQGJzCBT4ACBdxdp9+yGJz2Rzeyt7540F8mM0btJ3F5nL2VKP2jvw7bCmbw2OxRRcHlBThY2Nj+vr6WCxWQ0Ojp29A/AcBeuHU8hhWS5kk8BBH+MYd7jXPAoRLIvV+zgMQLu6mPjGFRKYtLS/vsGOWcnGbLoRCouy9RklZMxktfmlpmUhe3WBGOEiK8OXl5YsXL87Pz58+fRogXLwj/om98DUhvXEiQPh+xrMkbQcIF/Z0OxLHESgy6t3XN4u21uirsBo44t5r1PrNlfMzOMLq/m/CYcsIv3jx4tTMvHARcBz0wkEvXBKugDy7pgBAuLib+sQUHIG85xCxOcL3YKP23F3YKYPFJ6JJivDFxUW4Fw4QLt4FF56RLomDBgPpkqgE8nyiAruJ8PxOnskjtlISy+kJu2NW3nbURsAPEL5TKALlbE+B7SN8ZGQkOzubSqVmZmbOo9dYP0+kF46ZWehvg7YU4I/K3tHY5UTGBuEVmYlvKCQXBZHTX5LTXmwcRD4qmxkTbMmk/jaIXHqbUJ9XRWZtYFI5kVFPZVPLY9jtm9gDW8tq7BBeYBUs7fKJrAKXr6nA7iC8axYyzGId8GcIh+CXHDydj5BPbiLiCCeRqbPz6Nl5NIn83XC0QLAwODyGOOjZOdTMHOozvkEHvXDkXshBZPsI39JOZZhwJerTCOrTyC0FQpoNLl5XkksIqZa4u3qS5CRm2CHbnGBj1CW5RCQPLk6HkGwukrjmIe6uHjHbcc1TIonkQn+A8DWpAxJ3UIFdQDiHJzDNFeU3zPL4Oi7vxy/t5ADk4gg3sXFVuKJ59qpOQHjcysrKwsLCo6Knl1T1VlZWBAKBgbmT+k2r04paT8pefS5+bBvh3b0DLe1dsNmzc+iXVW8+VxNAvYgCu4HwN5gRQvdL2QnkiXbNjhIsFSM7Jq1a8r7Gr7/St+cFKugC6IXvILdAUYgCUkU4D+JTaMyHLXThzrdw/EwkY2CGiidS5CCQKHR4gywRhHO4PGVNEyKJsrS0tPxxprqHf4S5nQeMcBQa6x96Z2FxEU8gmd5yQ7zwLke2jfCsh0XHL94YGZtcWVl519Rm7eS9y5aD6sQV2A2En23KvdySJ1PhwNtUmbIHNubIuwyAcIQ3ILLjCkgV4TQGaw6NtyugCGNbOP5NEKOunzCHxstHwH1cz0oE4XgC6ayStod/hKaBzduG1pWVldl51NDIOIxw2P8uLy8/f1UbFnNP3B3vTsqnINzWxV/P1J7L48EIX15efvHqtYt3iFdg1MDQKJPF9gmKDom86+gR1NrevbKy8qq6zt033Cc4emxiandaJ6+1YLD4BwWlAoFg9Uc1hw6KiOdwuSsrK7uB8He4MeyrRNkJpN4q3c7HRCpWdkzCvkrEvckMBL3wwI126dhxpO23AqWKcAqNMYfGm+ZShbH9o3ggo6aXKB/8/tDSeQwB4gtEEA5B/N7+IR4PIlOoKpqmME4GR8aEEV5aVqF4w5BCoX0u2HwKwl9WvQmPuZdf/AxG+OjEpIaBNZlC7eh+f05Zd2Zu/soNQyKJ3NTWeVXDeHJqVtfUns5gdvUOqGqbcbl7aQ2cz3V31qsXiyMev3Dd3S+cQqUp3jBUUjfifNRzNxDegB5C+Z+WnYC8C5cdk1D+p4Xfhc93z7S/gbYaxnrYYLPR/UblLbVXqghnc3gYPCnqFflH2Baa1HY8jN48LCe9cBSWCK9tKYJwCpX2/FXt6mvvhQVVbTMRhC8uLmbmFrp4h9AZzPU89S6kfwrCX1XX4QmkCyp6CSnZ1k7er6rrLB28lpeXeRB07ML1/sGRm+aOAsHC9Mzc5WsGFVV1SurGUXHJ4TFJ9u6BNDpjF1onx1W8a2o/eUn9xMUb5nYe3I9d8F3qhTegh2b9FJrsD8sIMhGET/sqdDsfkxGrhBFOb37GGmrfemjDJdwE+4VviWr7KrNUEQ7PTUNT+cfCfzQXHSG6XiaLyf3RGhTyN52NwWCq37R69qI6OPJuTEIajJOh73vhg8Nj355RycgtfFT87NmL6s8Fm09EOPwi/OCpq9ZO3j19g9rGtgwma3h04oyS1szsvKGls2BhYXp2/vI1g86efgNLJxabg8MTM3ILIYj/uZosN/X29Q8bWKxKirRol3rh/W7H//Lzf5IRWCIIb7Y/on/olzJilTDCSc+iybVZ4oH0LAYVcJZafpf+Km2tkIqNUQcI31dU3lJjdwHhEF9Q1MU7GCBK8WORzDnKvvioDE8gNbd1tnf2IuPGbA6n9/3QysoKlUZv6+xt7+pr7+rr619N+Sx/n47wD2YHhsfdcvZdXFxKSM7SNLTRMbZ929BCpdKFEQ5B/Lv3szQMrI2tXSuq6+D5fZ+lyXJT6fLyMp//oyeh7SN8eHi4qqpqYmKCwWCMjE+KP1ALfxcOEL7pg4IwwofmJ58QGOJhcG4SHXSeUdFISioTD/TnjWAgfUtI22+ZdwfhXEhQ9p6nlcw6Hsk8EMQ4E800z2ENYuSQ3+LvwvcEKraN8PVax+FyFxYW1jvL5fI2OLveVSBdQgW2j3ALC4uenh5ra2s2mz09iwII3xTSG2cQRnjP7MSaK751z06gg87TimvWZA81rwogfE1lQCKswO4gHHYFbJ6AwRHQ2Kv/5ULyNn6OuDuRd+ESut3Pm23HEf55m7PPa98+wvPz8yMiIpSVlblcrngpIvuFT3ifCrr0p/Arf0q58d/zn3tqGzKQPuxxIvjyqlXlJgc25usunBVBeGz3kJKtk7KDa/z7MQTnCMKrLYOH3e+LkAkgXEQQcCiiwG4iHIGcfEcAwvc5QT9788XhK+ka6cHBwTwez8nJSRKE7wICJa8CQbjkl+xCThGEB79utEhKj+/7gd9ZGCqC8FR1m3aHOyIOGiBcRBBwKKIAQPiOP0/gSdQ994qXRmdtrMNefC757Cj9XAZsH+G6uromJia6uro8Hk+8FJFe+C4gUPIq9grCL1pYm8bfz0RTxHvhAOEicAKHkigAEL4xurZxlkpnLS4ufi73vb16N90vHA82G92esrt+1fLyCo643c1GFxcXFz7+QRC0KcLRIZcImU5bDdh4fUyUGj7LadOAjdPBRF0jZjhsGnDx+gjCMeFXt2oSIdMJE3UNG6u9qUn4LCfs7Wu4BINNTSJmOBBSrMR74Y55JQi84QjSC0+6YRV4USdRzQLln434btALR6QAkTUVAAjfBqQ3voQH8Wl0Jl+wIPt98eXlZcHCAo3O5HB5GzeKQmUsLS3tOo9AhVtWYHFpiUJliNxNSQfSt7TNCT7REDu3MNjJ31JY3ansSWQzjVNBYm4QasgswupOZcGU3GpKTtXGQWSnsrmJLVtFLo0i1ue9prA2MKmCxGykcahlqzuVbWwPfJbV3CWCcBF4iyAcF/gAG5iLDcwV9tQA4cJqgLi4AgDhIs5upw4pNCYKK+sLz6GwRBK+UDTyAAAgAElEQVSFzpNgaiGHB9Hon3PxmS2jbL9eQKUz4bX6hX/J0kL49IigpYa3pQBvNlpOZKzJMyTxEY6GaSgkFwTigh+Kuy2RFBGEj/Tyt2RSSw2PXHob9zYvH0dDDFgzUkZkUMtjWC1lIgaseSiyXziYkb6mSiDxExUACBf2dCC+gQJsDo9KZ1JoTAqNAYJMKsCk0pnstQZUpIvwipL5soJJCamJIDykrtnneXXw6ybhF8MIOIUR3uYQs7GbWxPhJbmDlU8wElqFIDyw5p3vi5qo5m7EEuEIgnB0QM4b67CNrQII31gfcHZHFAAI3wBa4BRQQD4UkC7Cta7Znjx6VUJYIgj/VvW655OXUa29myLc7PCFjZ3dmgi/dFZbVdFEQqsQhJ81NA2tb43tHhImNxJHED7hlXb0t3/Z2CqA8I31AWd3RAGAcPnw0aAVQIENFJAiwp8XTv/9iyMH/qHwKL1bEl4iCD945aqaq6d1anYmiowwEokI98K3gfCi7H51VZs//f7LipJ5SaxCEP6Nspp2ULjX0wrEEuEIQPiOUAcUsoMKAIRv4PjAKaCAfCggRYTbWUZqqNmY6Hvra7lIAksE4UeuacT1jQoDUjgujHDTwxdmfTNnfTPXc3zivfBzCjcMtFy1b9jFR5RLYhWC8CvW9sJmiMSFEX7kN3+e9c2c881azyrQC19PGZC+gwoAhMuHjwatAApsoIAUEZ4a96axkt1Sw8tIfCcJLBGEez17lTqFFWEkciiM8GfGPqnqNqnqNpiAnDV9nwjCh7q4EQGFTVWc+pe0wqw+SaxCEB5QVY/YIB5BEI7yz87Usk9Vt8nRdsAG/GgOOWIhQDgiBYhITwGA8A0cHzgFFJAPBaSIcEkAKZwHQbg4IIVThBG+qfsTQTiYkb6pYiCD3CgAEC4fPhq0AiiwgQIA4Zt8+Yb0woUfI8TjSC9cEgCAXrgkKoE8n6gAQPgGjg+cAgrIhwKSItzb23t4eLi8vMzOzq61vUu88cKbjeITDcF34Rv4X4DwDcQBp3ZKAekhvLd/OCYhDQ4oDE7cG4inuPuFE4jkzp5+R4+g94Mj4hm2l/L4WcWT8kr4WiKZmpiaE5OQlpadPzQyAfEFZS9rHhY+3V7J8FV0JsvJMxhHIElSCBZHDLmdMIfCWtp7rbesKQZHcPIMqaytn0djnTyDq9805OaXDg6PS1K+lPK8qnmL3M22zl4p1QKKlZICkiL82bNnycnJjx+X5ObmbrrAKkD4xl4YIHxjfcDZHVFAegh/+qIqKPLuxPTcxPQck8Vhc3gEEoVKY/Cg1W3C6QwmgUhmMFd31+BweQTi6qmrGsbzaGxiam5h6XM2h8vm8PBEMoVG/5gHIlPpRBKVQqPTGatXUah0HsRnsjhMFofF5hJIFBKFxuVBHO7qOmJEMhVe6BRPJCekZCdnPISd48wcStPQZnRi+vGzivMqejNzKCaLw2CxWRwujc7EfzSJQqPjCSQuD+JBAhp91U4miwPxBTQGk85gEUkUFoe72gQmC4Mj0hksHsSHjYH4AjKVhieSOVwel8cnkVetJZIocJNhAx4/e1X8tGIOhTlyTm0ejSWQKFye6L7pFdVv1W9aM1jsV7X1br5hDCZ7ZHzKLzQG4gt4EJ/N2WQxVGlgAIsnTkzP9fYPHzih2NbZy4P4VBqDQCLDxlC/X+yFy4MIJAr5462B+AIma9VWadgDytySApIiPCcnOzllFeFPnz4FCAcD6TvCGFCIVBWQKsKj4lPIFDqZQmcw2RYOXqHRiXZu/u+aO8Ymp2/oW8bfzzSwcMLgCIkpOQ4egX4hMRdV9YdGJywdvNx8w0fGp5y9QsJjkmxd/Z8+r+rtH1a4ohkanZidV2Ji44bBEU5d0kBj8bH3MmrqGt39IsKi7xlYOFXUvO3tH76opn8nMX18cuamuWP03VQ1HXNhhOua2sPuL/puamxi+oOC0ntpuRXVdQYWTmEx9zQMbIIi4w0tnV9UvukfGjW2dom+m2pg4UQgUVy8Q32Cbrv6hN1y8aPSGCY2bll5xVaO3gQSRf2mNQ5Pyit8aucWEBKV4BkQOTOHPnhSKTzmnrmtx9vGVsThuniHTkzNzqEwB04qhUXfM7FxLauoQc7CkYjY+xfVbrZ29vqF3tE2sm3r7CUQyWo65hwur7GlI+5+pkj+XTuMjE3OzS/l8vhNbV22rn7hMUkmt9wZTPZ1PQs337DK2nrPgEi/0DvufuE5jx7T6EwdE7uJqdldMw9UtJ4CkiI8NDT0fvJ9gHBxeMMp4F24VGkECt+GAlJF+AVVfQNLZ0Mrl5HxqW9PqxQ/fZmcmRcclVDyrMLCwavkWYWBhWNLR4+RlQvcqzunrIvG4tOy8ytr60fHp27ctGIw2e1d78+r6HX3DeqY2LPYHCyeeOLijbyip9aOPs8rXzt5BrPY3A/lV1S/9fCPuJuc3fN+KPR2IsQX3M94mJ1XAvEFPsHRayL8YeFTF5+wnEcld5OzX1a9Sc8tJFGoF1RXu+bt3X0JKdkZuYWwteeu6nT1Djh4BGKwhNl5zNmrOhMzs7omdnlFz94PjtLozGu6Fig07pCC8vvBESqNoaJl2trZ++WxywQSpb2rLyQqAfGtRlYuKAxutRd+Vg1HIDW1dvqFrHavhcPr+iYTGzeIL3jb2BoUcRfiC6g0hpqO+Wqvl8vD4onCmXcnzoP4lbX1tq7+cHX+YbH30nKLn748eOrq6Pi0jrEtrMyxC9eJJMr41OwFVX0uDxqbnFlzvc/dsRnUgiggKcJpNBqTyWSxWHQ6fdNeOC5el0OmsQj0LQVaQzG5wI/KoJI3C9TaTHKBP39mdtPAbChBdiojZtpzSFu2ipgfQK1KpWxqEoNKKQpktTzZ1CT+zCxvpHer25yIe3CwzYm4JiBFWAGpIvxOYjo8Hj46PqVwRXMejUVhcEQytbTs1b3UXBQGh8LgmCy2iY3r6lg0jS6M8LHJmRv6VnQmq72r76ySds/7IQMLJ3hDLUt7L2VNk5b2HgePQE//SAqVrmlg09U78KDwSfz9rJ73Q3FJq/3UtOyCzAfFHwafPQMixRHOg/i2rv5Fpc8RhD8oKCVTaRdU9VEYfPf7wbvJWZkPip69rIHtZLE5rj5hBCIZTySfUdLGE8kkCq25vVv9pvX7wZFruhZoLP6QgnLfwDCFxlDWMm3vev/lscs0OqNvYCQoIh5xpuZ2HvPoVYQfPXeNTKF1dL/3CY5GzsIRcYRTaHRVbTMKdfWdwmcJcyisorrRHAoL1x4cdbeptRNWhs3h3jR3wBFIcyjssfPXCSTK+OTM2as6YAj9s9ypNSuVFOGS71TWjBsjpFnLTmC8ydLvKqWR52XHpFVLHnqE9FX49rxABV0A25wIgwfEd0oBKSL8eTWM0o/vRNk+wdHufuF2bgFVr99Nz6J0jG19Q2Is7D3nUJjMh0UmNm7eQbcvX7uJ9MK5PCgoMt7NL9zM1r32bVNv/zCC8BeVrzUNb7E5XGVNk7vJWXQmy9DKxS/0jqWDJ9wLh+tFYfCGli5egVFahreEEX5RVd/FO9TRIzA8JonF5uY8evyxF14njPCe90MJKdkjY1MmNq6+ITHmdh4YHEEY4SNjk+Z2npGx962dfFBY3DVdCyyB+PzVa3M7Dxfv0MjY5Ok51A8Ij1ztScMhKDK+f3BUGOG+a/TCm7/vhbfBvXAMjqBjbMuD+B097+M/x0C6f1jsVQ0TV99wV9/wiuq3nT39Nk6+3kG3rZ18yBQajHAuD4qKT3H0CLJ28nlSXslgsa/rWU7NzCNtB5HPpcDOI9y676XX0GuZCicbsmXKHtgYlbYCgPCdwhUoR1wB6SFc3FsxPu6ihKTTP85lgw9ZHO6aW1YzmGwJO3PwHDekcDjCg/hrpotk2/hwvRI+9ONJFJrItRwuBM99E0lHDuub2u4kpiOHcKTgcXlsUgYSKDTR/Z4bmjuSM/PgzOJbSYqUtmuHdAZTvC4We/VOwumfZeaduEkgZecRnjvXWyFLoQYzpNJW8JY4LVNWvZp/b/v+JUC4OHhAyk4psJsIB54U4gsYTNaT51Xsj3PaEUEGR8bbu/qQwGKvzngXDq/rm/FEsnAKiAMFJFdg5xHegB5C+Z+WnYC8C5cdk1D+p8G78J0CFShnPQUAwiX3gyAnUGCPKiAthE/6nJIRZCIIn/c/Pe2rICNWAYSvBx6QvlMKAITvUacMzAYKSK6ApAgfHR19/PjxwsICBEGbzkjvdzv+l5//k4zAEkF4s/0R/UO/lBGrhBFOLLtDqEwRD8TyO6iAs9SyeGpZgnigvUzCxWpPdZZrtBbh8l/vlN8H5ciNAgDhkvtBkBMosEcVkBThfD6fwWCw2WwMBjOPXmNJReEFVgHCN31QgBHu3VcxXxI8XxSw3RA4NFQPEC430N3ZhgCE71GnDMwGCkiugKQI7+/vf/DgQVtbm6qqalfve/EKAMI3xbZwBhjhR99lKDfnfUq41JQLEL6z5JOb0gDCxd3UjqTQGSw8iYojkGU54IkUKn2NWeXrKUDbC42SZcGlahueRF3v6wlJEd7X15ednd3S0uLq6rrpQPqo50n1L3+h99Uv3U//dt7vM09tQwbS+1yPa360KkPjb8I0/SxxGOGOfRXZndWZHVXbC1kdVU+megDC5Qa6O9sQgPD1cLXt9NWV5uhMLg9aWl5eke2/5eVlCFpd+g35DGy9Vu+hRsm25FK0bml5mcuFqHSm+HeYW0Z4cHDwpgj/LFBcr1IE4etl+CzpMMKdeytORlkeCTfdXjgWYZbX8wYgfGfJJzelAYSvB61tp9OZ7IWFRSm6aikUvWlffC82Sgo67YEiFxYWaR/3ARL+AW8N4a2trcEhAOE7MK4AI9yzvzrgeabXk2SvJym+ZWlbDX7l6a/GuwHC5Qa6O9sQgHBhT7cjcQKJuizz/W8RFtHoq5u/bRD2YqNE2rhPDpeXlwkkqsitlBThki+w+vG78DPowHNbDaiAs6gAiS7cUs4feuGSFS5iNirgDCrgrEjimoerOQMlyxl4Dka4QmO2bmfpf5469K+H//GVpbZ2S5FuZ+mWwrX2IoDwnSWf3JQGEC7i7D79EEcg7zlaiPfbRHTYi43ac3dhpwwWHwKXCsJxcTrsyQH6UM/WQt0Dcr4PdaqXNNG1QSBPdlNf3SPleXG6GzYNzLePEIQTM2xZ4/1bM2moh5jnTX15lzzZs4FJpIku6lQvudCf+e7RpiatZuipgxHu9L6ydWr0XXNTy8RIff27psG+lsnRLYSp0dq5UYBwuYHuzjYEIFyEVZ9+uBdpBxC+U/iUhXJ2CeH4RMPpEUFLDW9LgdldS30aWU5krLehJ5z+CEfDNBSSCwJxwQ83dXmcnjcIwkkP3Ud6+VsyqaWGRy69jXubl4+jbWzVNjcb7a3GheRt2op1MwQ9nOkaAAhfV5/AB/v51O4gnAcJOmchpyL22TjmwRDm1STW7VccFIX/6byUwRIAwmUBY/vZht1GeFMVu6GSKSE1EYSnTmPvj6NSJjFrUlMY4XN+WRv76DUR/q6C0VjFltAqBOEpk5j746jUadyaVgkjfMYnY2OrGFUNcC/c9yPCN23FuqUBhO9vSK/7w/goy+4gPLuVezCAccD/R+FkNHMI+91+GDJI4m2bJI7wpaWl+sbWytp6gWBhZWVlcXHpTX1ze2fv0tISQpoXr2oZTBb8LrLuXcscCr2b79M/sRe+5979I7LLZWS3EW6k63npnI6EsEQQfuiqqkFUnEvBkwwUWZyXwgg3O3xhYy+2JsKvnNe7qeUioVUIwo+pa1ulZPm+rBU3KQtDRRA+4ZV29Ld/2dgqEYS7nL62cf51zwKEA4Svr8AuIHyewr90hynCb/jQJo/N5G40i2rbHP2MF4ognMli3dCzfPq8MvNBYUDYHTabo2fmUFBSFpOYVlRaDiPk2fOqo+euzaHQLBZbVdus5OnL0OjEwtLnuwaYbSN8YWEhKDLexTsk51EJAPmu3a+NK9pVhFc/xf39r0f+938fe/JwRBJeIgg/cOWqoq2jeWJKphQQ/qJoRvu6/W9//UX1U5wkViEIP6iofN3T162kDCB83UeK9XECLtl9BaSKcA4PwhEpERXkNfl9wJ9xPIzePEyYQ+PlIKBxJHiHMRGEUyi0Zy+qVlZWOByu6S33yelZR4+glZUVLpdn7xawsrJCIlHc/cLPK+vModAYHP7kJXUUGvvmXXPsvYyNPfUOnt02wh/klz4oeMLhclW1zYZGx3fQJFCUJAqw2Rw429LSEpFEgeO7inB3h3tXLxnoqjuYG/pJAksE4UfU1CObulIm0WvC8hN74WpKptrX7dSUTJNiqiSxCkH4RQubpLG5TQfSQS9891kFalxTAakinEJjzqHx5rnU9RB+MJBR20eUA37DTUBhiRBfIIJw2KsuLi66+Ybl5j/uGxi+HZ+ysrICQXxTGzcul6dhYDM4PAYjfGFh0Sc4+rSi1lkl7dGxSUmc+I7k2TbCDS2dB4fHVlZWbsclPyl7tSPGgEIkVKD69TttI1v4/UvV6/pzyjpoLG5lZWVXER7gkdXwavVFeKhvniSwRBBunph6f2x+TX5nYajCCL+rZmF/XMn+uBLKP3tNLyYykD7QyTa96dNYyX5dTkqOrZXEKgTh9jkF65kkPJA+65vpcvqa/XElrzM3MAG5a1qFDKT79VUT7z6N0rF2VNSM0LIkxpVuNcz2DYPpbGuKDBKlinAqfRXhdgWU9RD+TRDjzXv5QTgGT1oT4VwuLygiPjw6kcvl9Q+NRNxJWllZ4UGQnat/eUWNjbNv1et3xy9cL37yYmB49LqeBQ5PbGzpsHdf7aPvzt+2Ea5wWX1iamZlZSUtO/9BfunuWAtqgRVgc7iGVs439C0fFT07dVnjXmrOwsLqZItdRbgkgBTOgyB8A1KKIHxTNy2CcFmbkW7Y+aR0qu9Twr2xFoDwTX8G+zODVBHOg/gEMi3t7boIV4igd44R5zEEOQhYAgVeplSkF87n8w0sHD+618XFxcXZebSGgfXi4iKZQvUKiGzv6s16WJz1sPiQgnJy+sPWjm6f4NuLS0sEIlnLwGbXWLVthN80c4B74RF3kiqq63bNYFARrACTybqqafzV8Svx9zMRTQDCZeujMp3O0oyZ3k8JkWPNAOH7k9CbtlqqCIfnlNE5Av0M1pod8dvVHC4k59PZRsYmD55S0jWxN7BwcvIMWlhYjL2XbmztauXg1T84grjdy9duzqMwTCbL2SvYwt5T19S+6vU75Ky0I9tGeNnL6oTkrDkU5vK1m3MojLTtBOWLK0AgksteVsMfO8BndwDhBAJhYnpWfFKo8E5l4Lvwjd0rMpDuNVhXOMzPH9pmKBjitxEYAOEbq71vz+4CwiH+6kfhivGik9KNctjyx+81B9LF3S6XxxNPRFKYLPbC4q6usr5thC8uLYVGJ1o5er9rakPsB5HPq8CnIpxMJispKT2vqAQIFxntRz4qkwQYCMI9B+oeDvBz+7cZHvTzW/AA4ft6/ZYNfm+7g3CILyAy+LG13Cv3WQfDmdrZrPI+HoMjb/1v2OOJDKR/Xm8uYe3bRjhcvvAH7hLWCLJJT4FPRXhJScnZs2f7B0cAwncQ4WahGTqukQ53S7bKcoDwDQAGTu0awsW9gbym7EOES49GoORtKPCpCI+NjVVUVETjCOL/RMFAOrU8htVSJgk5RHrhKpbeGV3MPx08ce8ddksUBwiXRO19mwcgXNxNfWIKQPg2qAMu2UEFPhXhBQUFp0+f7u0fEv+XIIJwPGphpIe/pQDPSG+jc2oorA1CHZVNaCyiFAdTC15T82s3DiIz0lFTW7aKXHqbWJ9XT2VvYFINhdVK51DLY9gdLze2Bz7Lbuv5brPRjwPpR5W0rpq5azqH5/TxAML3LXF3vOEA4eJu6hNTiGTanlunjEZnbtzqvfhcsoNQ3FtFfSrCyWSyvr7+i4oq8d+EMMLRoVdIDz1IeVsL+HtG2Bh1Sa7CJxhgYzQkyplkgmxzgolUleQSkTzY6Bu4u/oiiWseYmM0CMnma54SSSRmOQgjXNXKZ0vkRjKDXviOY0+eCgQIF3dTn5jCYHF4EH8POX3BwiKdAfYL30N3bCNTl5eX8Z++XziTyZzH4MT/JSAIf40dIWJkKJAIU+rtxWg6XqasImLHfAeqPQdfz7K4owTSLIu7vdBMwqq05KNxODIGDwJQQFiBEey8YssjDJEwPT0t/g8WiydzeVyBQLC0tCTes1xaWlpYWIAgCE8ki1+7n1NodCaDxVlc/GEXk42c7uc7t7S0zObwyFQ6j7/J1EIag7XLk+Q/nyp7u+aFxTUeyHZ4v/DEqTbdrlIQgAJAAVlQwHmgEgcQvhnDtvpEwmJzqXQmhcaQ5UClM5ksjiRN40F8Gp3Jg/hLS8t7G3Hya/3S0jIP4lM/3iaRe7qTCJ+dnSWAP6AAUECWFEChUKAXLuL1wKG4AjQ6C4sno3FEEGRQASyeTF3nhchOIhyFQrHoDCaFBgJQACggCwqwqHQ8Hg8QLk4skAIUkA8FdhLhGAyG+rJ1OipvMCRzKDQLfacQG1ss7YC+UxSlbj4dlQdXhIop7A1MnYvO36DekbDs2ej80bCcDfKsdyrT2BU5hYop7AtMm4x8iKQIR2aj82duPxJOwcYWz0XnT0Y+EElc83A8IncoNHPNUxImYu4UDYRkzMcUYGOL52MKJiJW6x0NyxE2eDY6vzcwFX2nCBtbPBX1cDg0Cyl8PDx39nv7x8Jz0XeKhkOzhkIyN9YWuRxEZEEBQlYFCQ/ehW/yMlg+XDloxf5UYOcRHnLN+N9/8r9+/v/9RP/I+W17sdBrxs3eCZJc3h+cbnDsIpxzPqbg4t8O/f5nv/ztv/1i7nv8iBfiflnzuX3osT/+t/gpOOXWWVUYeCIZMHeKXC6pw4njEbkHfvOnX/7Lv/7XT3/e5X9fOOf9mw6VTpGxWlb39GyF07GxxQWWPsYnLokkih+W3gr8yy9+/V//+u/3DRyavO4GqxkJ53ls498TkCKcsmbc7bLm4d/99fSf/zEVlafy1VHj41caveJ/8S8//ef/93/Uud+BL9H+9sy3v/ur+xWtVp/E3//rL/747//R+X1bLvztoJeSDja2GBVT+Nt/+8X7oPQ//vw/Vpv8k582eMavWSNIlDUFAML3p1sHrd4/Cuw8wgNUDGK0rLCxxWe/+KrDL8nmjIrZKaXJyIfOF9V1j5x9ahtkevKK0YnLfUFp6gdPWZ5WMjt5xUJBKdXAqdUn0fD4RavTygPBGX/4+a+cLqo/MvcyPHbR+oxyh1/StQMn1L4+EaRmaHtWNdXACfGVhZY+j20C4MM4besbh06hYgoqnSKf24d6KelqfnP6rXus+Skl4xOXW7wTHpl7Gx2/dOi3f3nlGOGlpPM+KE3v8Dmj45fafZPsz18zP6VkcOzCQzPPn/yPf8oycQtSNbQ9p5Zu5IzUlW3slm3shtR165zqXHR+sZWfxqFTCbq2t86q6R05V+Uc9bdf/cb8lFKFQ9gLhzDni+q3zqqafmxj2DWTN24xyTcdY7Wsrc+o6B0599b9zs2j51ebHJJhfkrR6Hu6eynqeCvpNnjGxevYWJ9R/urXf+gLTHO8cMNSQbnIylfhL1/ePHrhtVu01iEFg2MXGj3jTU5cNjt5pcwuxOOKdsv3jz5Hfv9FgaWv1jenewNTnS+q6x+5YHDsovNF9fAbpq0+idjY4rHwnJN//kfodeMHZh7RmpbldiEFlj5mp67ADTz1538c/O2fsbHFd7Ss/u2f/6UvKE3l62NtPvf0j55P0rdDNAERWVYAIHz/uHLQ0v2pgKQI5/F4S0tLNBptfHx8Zh4jLhb8URn1ZWuAisEff/arL//rjy6XNFp97j0w8zA+finlpuORP3wxGJJZ6RSZbuiif+T8QzPPv/zi1xMRDy79/ZvJyIdqXx/3U755W93C+MTlh2ae5774+rVr9K9+8tM7WlYqXx3Lt/A+9se/oWIKNb85Ha1p8cjcG/abmDtFekfODYRkwIcO568l6N56civwi1/95tqBE+e+OPDSIazdNynVwMlbWS9G01LjkMJoeM7No+ef24dcP3gi3cDJ7pya62VNhwvXD/zmT13+yV/++g+1rtG//unPxsJz1A+dStCxRZ4P5mMKfvezXwyGfDe47amoXW4fgo0t7vBL+vo3f7RUuFrjcjtJ397jirbO4bOltwJzTT0emHmofHVsLDzn/N8OTkflffnrPzy2CfBS0p2LLrA9p5Zm6OR0QT1Jz17727PJNx3++O//MRqWDTekPzg9SM3wb7/6jb/KzUcWXmYnr4xF5KYYON6/6WB1WjnihlmTV4LnFW0vJV0LBSXnS+pf/9ef4rVtntkGN3rdhQe60XeK9I6eO/PXr4784YupqLx8C2+db88d/M2f7c9f0z18Ns1w9bnkmW3Ql7/+Q76Ft/qhU7Wu0We/+FrhL1+e/NP/hm049ed/KP3j8EMzzzN//cr8lFJfUNqffvGfSv84/Puf/arAwgfOA/4r4woAhIt7KpACFJAnBSRFeHBw8MLCgoWFxaVLl96+axKXQBjhYddN4Reukerm9/TsDI5diNWyunHoFDa2WOvb05HqZhf/fijTyOUf//n76ag8g2MXpqPyVL8+FqFuGq9jE6RmlG/hfe6Lr186hP3lF/+ZYeRqc0al2NpP9etj2NjikGvGT24FHv3Dd2Pgb91j1Q8qYD6+ysXGFpfaBHz5699/GC5OMXA8/8Whc18cmI3OD79h6nJJ3f78tfDrJjqHz3b631f9+hiM8FxTd/tz1+7q2Pop6x/4zZ9mbz9S/MfhCqfw//hf/zYYkuFzVbfKKVLj0GnYTb9yirA8fRVx2Y/Mvb753Rf1HrGuFzU8FbUtFa5mGLkEq55xd/oAACAASURBVBoGqNzUP3o+38IbRrjmt6cnIx/cPHp+Ljr/0G//XGzl56Wk66ukZ6lwdSLigbeSbrqhi7eSbpqh06W/H0IKtz+nVmjpU+kU8X//n/9XprGL/tHzZXYhWt+eTjFwNDh2IUrdvMo5MlDVwOeqXsg1Yz9lfW8lnTtaVme/OJBr6jH08X32dFTeb376876gNPNTim0+9/ItvPWPXLh1VjXFwDFQ1eCenl2+hfdwaJbSl0eGQ7PUDym8dAi/f9PBU1E7zfC7EY5Tf/5HmqHzv/7zTzyuaDtdUl8dNTl0ajw8N+y6ybWvjyOmgogsKwAQLu6pQApQQJ4UkBThBgYGaDT62LFjZDJZfI03iC9AEF5g6fPMNgj2ay3eCTePnLc6rVJmFxyjZYmNLX7pEKbz7RlvJd0al9tulzXnovMTdG3novOjNMwHgjNszqq6XdJExRQ8tQ3yuar32i3a5OTl2xoWPQEpURrm2NjiFAPHGwdPpn/sRGJji10uqr92jRb2oRlGLuqHTpmevFLnfidYzQgVU9jhd//mkXPeSrol1v4VjmGa3yjYnbvW4p0QrWk5Fp7jqahjc0blfVC6h6L2fExB+A3TNt97cdrWRVZ+SXr21w+eyDZxh8v3V74pPL8MHVOYoHNL/dApl4vqU1F5lgpXXS9rGB+/PByaVet6O0Dl5mvX6FrX6Dht69nbjxJ0b6FiCj0Vdd55xD008/JS1NE5fFbn8Nlm7wRzBSWvK9qjYTmh102QhnQHJBsev2hw7ELprYCRsGyHC9e7A1LMTylanVZJMXBs9Um8eexCb2Cq44XrThduvA9K81DU1v72dJldcLCqYYffdy/mC6x8dA6fCbtugr5TWOd+J0H31kBwhtHxS+5XtEbDcryUdEfCsuN1bNQPnap0ihwJy7Y5o2J3Tg0V890kxACVm+2+9/yVbzZ6xWeZuI2F55iduqJz+KzzRfVO/2TEVBCRZQWkhHA2h8tkf/fNMYPJ5kF8SXzi0OgEhUaXJKd4HjKF1vP+u3WduTxo2+WIl4ykTE7PkchUOoPF5UFwIgZHQGPxSAZ5jTBZbAqVDgcOD0LirO9vMdxwOoOF3PQ1pRgZn5xDYdc8JZLY3TfIZLFFEsHh9hTYAsKJROLBgwcpFMrGCN9NjwZPpd6dGpG+/prVWZ2+Or3O1PQ184NEoMAuKCAlhD97Ue3gEQR7nKCIu2su1yjij3gQX8PQprWjVyRdwkMldSN3v3A48+T03OEzqjvLADKVfuqKJhqLd/IK7uodgCvqGxjWMrRhc7gSGrlHs91Nzvr6hOLXJxQPnlTq6huA41+fUExIyRFukYqWaXpOgXAKHCeQKEVPnkN8QVRccsnTl+IZkJS6hhb4OczKwRu11hKfSE4QkVyBLSCcx+P5+fmdP3++9k29eAVIL3wXHBOoAigAFJBEASkh/OmLqi+PXU7Lzv+wYHhgePw8Bvei8s3UzDzEFxQ/fcnh8goelz8uexUZl9za0RMQHldRXbeKcAPr5IyHgRFxzW1dEF9AIlMTU3Pi72dhcITZefTjZxWx9zKoNAbsW8YnZ2IS0lKyHhHJlIbm9m9OqwRFxMOnJqbnvlFQRhBOJFESUnLCY5K6+wZfvHrd1Lpa+NPnVf1Do5PTc7H3MtJzCsgU2tDoRFlFTVxSJofLe1j4JCQqob27D/Fj9U1tHv4REF/g6BHU1dsPp3N5fENL55HxKTaH+7axVaRXily71yPx9zNr3zaRKTQyhTaHwl7VNJlHY8kUGoP5Q0f5/eDIaUUtQytneIiiub0rMCIuv6SMxmCGxdxT0jAufvryVc3bts7eu8nZM3NoJoudkvUIiydm5xWH3E6ofduIw5MMLJws7D2HRiYeFj79WBcm/n5WQkr2PHp1xe5Hxc9yHj1OSn+AxuJ5EL9vYHivC7s79kuK8JWVFXjZZAaDjsWTxI37AeHxJdj4x/IcYouxct9G+b6D+6V1JdJDeGBEnJKGce/AMIzwgPC4lvZuiC+wcfJlsjmahjZ5RU8dPIK0jW3bu98ra5pweJCGgbWtq/+7pnZlTRMsnmh6y+1eWm5+SZlf6J2u3oFTlzXKK2rhLi+JTFVQ1Cp5+vJeau7t+JQ5FObYheu1dY2w2xFBeEX123tpuQUlZSraZlW171x9wngQ38TGbXoWpahuVFj6PPpu6v2MvOo3DWev6rxtaB0Zn7rl4tfU1mXp4IX0sHMePX5R+VoE4RBfEHcv411z+zwad03XYnoWJe735CAl/n5mem7hu6aOhuaOeQzuopp+zZvGd00dyOMUxBckZ+Y9KHhi5xbQ0NJBotA0DKxb2rttXf2r3ryrb2qzdPSenJkLjrqbX1IWejuhqPT5zBzazNa9s6ffOyiqpq7xgqre3DwmNDoxPXf1ccrQ0nkehXX1CSsoKc/ILdQysmWyOZeu3SyrqPEKjIq4cx+LIzp4BMqBtrvQhC0jHIKgjQfSOUMzjHGivAbuPBmfWk4bnZsangYBKCDLChAnUNJDeEJKTv/QqL6Zg1dg1DwGJ4JwY2sXiC8oLX/lF3oH4guMrF1pdKaGgfXbxjYeX+DkFVxeUXtBVV9Jw1hF2+zKdYOu3gF33zDE3w2PTV65bsCD+AQSxcjKhclin7ykMTbx3WYtIgjvGxjRNLRR1jK9csMQTyRbOXh/WPHb0z9yehZ16rKGqo65ipapnVtA9ZsGeByYTKEZ27iq37TOLylHakxIyW7p6BFH+IOCJ89fraKdutmWnUhRey4Sfz8zISX7eeWbV7X18xjceRXd0rLK55VvyBQa0hZtY9uQqIRruhb27oG1b5tCo+9BfAGDyWaw2HMoLPyOA0Y4jkDyDop68rzq2YvqmTmUkZXLNV0LhStas/OY1Oz8iuo6iC+AEW5ktfoj4fIgFS0TPJFsZusO8QUNze2mtu48iC/hAu+Ihfs2svMIZ09gW2p48hrmR7iE9BeN3SMnM56AABSQZQXejMxIFeFcHhQVl/zlscvzGFxY9L3kjIej49M39C2ZbI7prVV3/KS8Eka48fcIv3s/e3YOrWtiNzw6cU3Psr6xbXYe87axtat3wDswCvHCKAz+5CX1/qHRt42tXoFRayJ8dHxqehZFodJDbic8KatsbOm8csOQRKHG3svw8I8se1mDJ5KV1I2GRydGx6fau/qq3zRkPSxenXg7j25u725s6Tx67hoOT4Qrzcl/DKPawT3wVW399CxqehbFg/hxSRn1Te1MNqf6TYPwwDJiqhxE4u9nwq82IL5gHoNT0TYT7n9DfMHo+LSqttmbd81Pn1dpGFj3D41a2HvOo3H5JWWdPf04PMnC3pNEpsII/4B2Z68QM1v3qZn5BwVPwqITh8cmL6jozc5jMh8Wp2TmMVkcGOG3nP36h0ZbO3qu61nQ6EwLe0+IL2hu6zK95c7icNu7fnjNIQciS68JUkF4czW3+imutozQWMmSM5YLI/xESvHh2KwTaSWy7MeBbftWAekhvPxlbVL6A9grGVu7ojH4voHhQwrKGgbWlvaeLDbH0sEL4guevaiGX2Bb2HvSmSwdEzsH98BvFJQzcgshvmByZv6qhtHhs6r1jW3dfYN+ITHCbq7uXcvZq9qq2mZTM/NMFvvsVZ3xyRk4w9T0/LHz179RUP5GQTnnUUl9U9s3p1WsHL1VtM3IFFpLe/fXJxRnP65d0dDccV5Z98g5tf7B0dq3Tbn5pR/62TQ687qe5eGzqhk5q2bA4V1zO2yAg8eqhXBgsjjG1q4jY1MoDP7IObWZOTSSX54iiSm58EsQiC9AY/E39K1EEB4YHpeVt/r0A/EFgRFxPe+HMnMLDykoW9h7wn1lExs3R4+gsJh7RaWr89pSsh5d17Pk8qDBkfHDZ1X1zRy1DG/NobB9A8PfKCjX1DWa3nKfR2O7+wYUrmhcvmbQ3Te4+grG2QfiC1rau60cvfEEkrG1qzyJLL22SAXhlaWYn/3bL4O8cq9eMnhdTpIniiMI/yYi+WeHjv3JwPpXZ64cv1+wbzkBGi6zCkgP4Wv6Iy4PkuTrMpbQBG8uj8/hfvcFl3iZyMdd4qdEUjhcHlw1jkCycfa9HZ+CWLKeVWwOT7gQIolyUU0fBj+S3vN+yMjKhfHx8ycuT6IP55Br5T7yUfMfFp9HBBdpOJfHFz4lHIf4At7qQPraworkFCkWHCIKSAvhP/2fPw/1fXT1ksHblzS5RPhfLZy/9Aw/mfHkTwZW30any6wfB4btWwV2GeGIT/mMERqD+X5wdHsGYLAE4be/EF+AxZNIZOr2SgNXAQV2RwFpIfw/fvW7F0Uzdc8p8sTvlhoe0gv/yvf2r84qHr374FdnLh1NeLhvOQEaLrMK7EOE747TBLUABWRHAakgvO45xdYiXM7gDTcHQfiJtJI/G9/6zwvKf3fyP5leKrN+HBi2bxUACJcdPwssAQpISQFJER4ZGQlB0MLCwqYflYEZ6fuWGaDhMqUAQLiUnCYoFiggOwpIivCUlJSJiQmAcKQXLlPOGhgDFBBXACBcSn6WzmQTyTQ8iSrLgUCm0RgsyRXYE42SZcGlahuRTKMLLZYnfFslRXhNbU1HRwdAOEC4OCpAimwqABAu7Ol2JM6D+FQ6k8PlLS0vr8j239LyMo/Hp1AZ6035RgTZQ42SbcmlaN3S8jKbw/uwZpH4RH2A8K2tQgMQLpu4AlaJKwAQjlBqpyJ0JlsgWJCiq97popeWl2mbrSu32qiFvdSonRZpz5QnECyI98UlRfjk5CQajQa9cIBwcVSAFNlUACB8p8iNlEMgUZdlvv8tQiQafZPhdAKZtucaJdLGfXK4vLxMIIl+5SgpwpFtTsB0NrDAqmwSC1glogBAOILenYrgCOQ9R4tN34jvxUbtubuwUwaLb1ACEA4G0sFi7/KpAED4TpEbKWcv0g4gfKfwKQvlAIRvDdjin7aDgXSRrh44lFkFdh/hXEjwZoRnnMs+Ec38NoKpm8l62s3jQj8sw4mwcI9GAMJlAWP72QaAcIBw+exxyixHP6Nhu4xwJldgV8L+NohxwP+HcCiQcS2DRaCvvQ72ngP5mgjncnnFT18gXGlq7Xzb2Lq4uIikfN4I6IV/Xv13tnaAcIBwgPD9osAuI7xjGjoe9gO8hUFe0sndc7Re02BxhBOIZHff8DOKmisrK0wW67qeZW7+45iEtMjYpJ313dsubdsIr61rdPYK9gm67eoTGhgeRyCSRGxYXFwKCIvl8wVIekhUAjy5fWFhIfR2wjwKg5wCkR1RACAcIHy/AOwzdn9lpGqpIpzLg3BEyhwajwTDbKowtoXj1+/TkGx7MYLBk9gfN1gTR7ilg1dIVAKMcDyBVPi4/CPL2aY2bjvisj+9kG0j/FHRs9LyypWVleXl5fikzA8gX1lZWVhYIJEpVBr9Y3zxup4FjwchRmob2cBE5/MF2sa2o+NTyCkQ2aoCEMSHL1leXmYyWXAcIBwgHCB8vyggVYRT6UwRGH8bQRfGtnD823C6SOY9d4jGEiG+QBzhbA5nZGwSRjjsZBcXF/3DYnPySrbqsqWU/9MRvri4GJ+UGXnn/srKSlBEnIdfuOkt9wcFpQsLAOFSumkr9Y2tZrbuLBZ7ZWWlrqFFWdMEhyeurKwAhAOE7xeAyUhX+DOascsIPxGzLsKPRu59hONIayJ8ZWVFGOE8HhRxJ8k3OJrL5UnLu2+x3E9BuJq2mbG1yzcKyrYufgwma2h4XEXLtLu3v7au8dRlDRKZKtIL1zK04fNX+458vkDLyHZsAvTCt3i3vs/OZLG1jW21jG6VV9QqXNGMuJMErykEEA4QDhC+XxSQKsJ5EB9Pos5jCEh/2i5/3YF0vbS9PpBO5nB5myKcLxAYWTnfS82RqZVSPgXhj59VLC4uVta8NbR0XllZqa1rvHzNIPpuakxCWlxSpjjCja1deNDquDqXy7uhb4X92HH8nkrg/1tTgEqlX1DV++r4lcjY+8gvavsIZ7PZampqoaGh1dXV8xic+FyP2dlZDAZDfdkKdir7jB0vUDVQAFFAqggX9wAjWP7pSKbw+DkSrxpc5Z8cBPGBdOFeeFfvwMFTV81sPaydfHyCo7fmsKWW+1MQDr8LX1lZ8QuNCYtJRGNwqtqm/YMjzW1dbj6hHC5PUd2o9/3Q0Mj46PgkXyAIjozPfFA4MTX7ovL1TQunxcUlqTVrXxSMweHzip4KTxjcPsLb2touXrw4Pz9PoVB63w+K/2sECEdcJ4gABWRBgV1GOMQX3K7lnvjxpPQjwQzzAjZPLvi9Xi9cxmGybYRXVNW9rm+CWzePxuqZOcyjMG8bWjUNbaydfMYmp5eWljQMbOBgaOWCxRNIZKq7f/h1PUsnz2AMFi/jyuxF87aP8Obm5osXLxIIBAqF0tM3ABAuCz4a2AAU2ECB3Uc4DxJ0zkDWBeyTd5iHI5lGD9hvRsDSLp+ZFNtG+Hp2Ly4ubbxR24LMfBO/XhP2bvr2Ec5gMGCEczicORQWIHwD1wlOAQVkQYHdR7i4W5CzlDUH0mWcBzuOcBlvr3ybt32Eg21O4MVWwQKrsgAnYIMkCgCE7/gDBEC4fANS9lsHEA5mpO+X+diSQE6+8wCE7zjCiXtwX85N9wvfi88lss9aKVkIEA4QDhC+XxQACN9xhDOYbEhoPVEpuekdLHZxcWnTgfS9uAn6Dkq0h4paXl7GE8F+4TWfRHEwkC7fPVd5ah1A+I4jHOILaHQmi81dWpL1z6WWlpY5XB6FyuBBm+wxQ6WzZGdTlj0E1N03dWFxkUZnifyqwX7hWyM6QLg8QU6+2wIQLuLsduqQyeaQqQwShS7LgUxl0D+OGWzaah7EhwfbkfVDdh9OoMaNFVheXob4AiqdKf5ABhAOEL5fBpblG9jirQMI35ReIAOiAJXORONI8xgCCDKoABpHotCZyM0SjgCEA4QDhMunAgDhwp4OxIECcqkAQDhAuHwCTLxXut9SAMLl0mWDRgEFhBWQFOGRkZEQBDFZzJCQkJrXb4WLgONggdX9RgjQXhlXACBc3E2BFKCAnCkgKcKrqqrev3+fnJxcXFw8M48WVwEgXMYdOjBvvykAEC7upkAKUEDOFJAU4cvLy4uLi+7u7oqKilU1b8RVAAjfb4QA7ZVxBQDCxd0USAEKyJkCkiL85cuXXV1dSUlJxcXFKAxeXAWAcBl36MC8/aYAQLi4mwIpQAE5U0BShMfExPB4PDqdHhERUd/QLK4CQPh+IwRor4wrIFWEs9hcLJ7IYnPEXYH0UlgcLpcHfUr5s/OYNS+fQ2GES+ZB/LGJaRaHu2ZmeUrkQXw6g4V8bQwvPydJA7k8/tTM/Ho5mSwOBkdY7+ya6TyIT6bSSJTvVh8bm5zhQRLtMU8gUZCr4JLxRDKRRFmzFvlLlBThP97mhCQuBEC4jDt0YN5+U0B6CGcwWTctnE5cUr+qabIeFMVdxKenZOUVv21o3Wo5PIj/uKxiYmr2w+IYDu6B4pfPzKGtHL3ZHJ7wqfCYxMraeuEUuYxzedB1PcvpORTEF3C4PNNb7hI2k0Zn6ps5rJd5aHTizr2M9c6umf6o6NmpyxqnLmvcTc7m8iAP/wgO90d3ZM2rOFzeDX2r94Ojwmc7e/utHL0luVz4qj0a3x7CyeKtBQjfb4QA7ZVxBaSH8Nz80tDbiWwOr7S88vmrWogvqGtoSUjJ6e4bhPgCDJaQnlPwuKwiN7+UzmAlZ+ZRaHQimfqicnUOTVtnb1L6g9q3TVwe1NbV19DSkZT+sLmtiwfx51DYrIfFmQ+LcITVTkJ9U9vd5OzOnn7E2yRn5NW8aUAOG5o7ElNzmlq7IL6g9m3Tm3fNiak53b0DEF8wODJ+L+3Bm3ctZS9r5jE4VR3zgPA4CpVuae/17EV1StYjFAaHlBMcdTfrYTHEF/S+H4q7n1ny9CWbw0Vj8W6+YRBfQCCSxyamkczyF7mXlvukvBLiC0rLX8Xfz6LSGAUl5Wk5BfDDGXw3sx4WU2kMFptT8qwi7n5mR3c/lc7QMLCurK1PSn8wOT1HZ7LKKmpW7z6OUN/YNjAyHhWXwuFClbX18fezXtXUc3hQXUNrU2tnaXnlk/JKIpnK5vJSsh4RiN/RRM/UHosnYvHEO4npNAbzRdUbNodX8Lg8OeNhcsbDju73HC6vovrN/YyHI2NTyF3o6x9WvGHIYLIfFD6h0hhEMjU5M4/+8RFzZg7N5vBaOnpEHs6Qa+UjAhAOvgsH34XLpwLSQ3h7d9+VG4bJmXnDo5NsDndweOymhdPbxrYLqnoT03MXVfXvJmffT3/49QlFLJ741fErs/OYialZa0ef4bHJQwpXq980GFg4Do1O3M/Iu6phXFZRo6JlSiJTHTyCsvNK4pIynbxCBofHdU3tG1o6rlw3mJ777hMYYYR3dL+/oKJX19CqZ2pHpTF8Q2Jumjs+fV6lZ+bwYRhc19Q+r+hZYHicrqk9lcYwsnYpLC1nsTlXNYzzCp96+EfYuwfAHpzN4RlZufT2D0N8gaGlc3v3e3e/iLeNbRBfYHrLDeILHhU/9Q2JQYaa5cPvC7diZHzK1MaNw+WdV9Zt7eiJSUiPvpv6qOjZt6dVyFS66S23R8VlcUmZ7n4RtW+bHD2CXtc3axnaUOkMFS3T0OiEjAdF1o4+eCLZ2NoF4gv6BkbCYu7BCB8Zn7qhb/mmvvmSmv7k9FxIVIKeqX1H9/uE5Ow39c3DoxPHLlynM75b9NvRI8jGyaf6TQPh4xi4laMPlwcNDo83tXaeuqJZ19BS967Fzi3gReXrbxSU4Yc8iC94Xvk6+m7qh7XrHT2C+gZG3ja2GVg4QnxBROz9rt7+qZn58yp6M9//foRbLTdxgHCAcPkEmIx3kXfBPOkhHOILxqdmXlS+1jK6FX03NfZe+uXrBgaWzscvXM9+VHJIQZlIpuCJ5MvXDEQQXvKs4tvTKgaWzorqRkVPXtzPyHv6vAriC2xd/VAYHIxMDpdHIFISUrIvqukbWDqfuKQOd/QhvkAY4fHJWScu3jCwdL6gqvfmXYtvSEz1xw66taM3lwdZOnhCfAGLw9U1tedBfHv3wJb27g99RAu71fS2zl4TGzf45TeTzdE1sYc/lI27l6FpeCskKoFIXn0ja2ztCg8vI5iRG78v3BAqnamsZVpRXXf0nBqNwdQxtp1HY7k86PiF622dvaraZhQanc1ZnfowM4928w03tHI5fEYVRjg8GcLY2nVNhJPI1ICwWCNr1yNnVXv7hz8gfHxyBuIL+odGXb3DCh6Xp2UXIJaQqbTWzt7YexlqOuZEMtXK0Qc+dScxvejJSx7E9w2JVtO1MLBwPnBSqb2rDz5bVPriwaNSiC9obO1MzXoUey/j/cAIxBek5xTUvm3iQXwimSrHj18QXwAQDhAOEC6fCkgP4YWPy8te1pDI1MwHRS7eoWUva27HpxDJ1IbmjqmZ+aPnrrW0d3d0vz9xUZ1Ipnx1/Epja2ddQ4upjVtTa9dNc0ccntTVN4DFE+9n5MGj606eQSgMztLBe3xyprd/6HHZq+eVr0NvJ5LI1MbWTvz3Y63JGXklzypQGDwKg3/2ovpDv41AorR19tLoTN+QmLp3LfDTAA/iG1g4vXnX/LDwKYxwR4+gspc1XB5k/REMXb0DCMLZHJ6hlUv/0OrL1Jb27tk5tImNa86jEqQXjsERRsZ/GLlFkCNPkdjEdDUd89SsPIgvcPEObWjuGBmfOqSgPDuPvqZrMTA01t03WFT6PCw6MeR2wiwKo2V4C0Z4c3v32MS0ma07kURR0zGfnkU9LHyC9MKr3zQYWjrPzKP1zRyGRiZCohJmP64pQqUxlNSNrRy9kb3MWWxu9N1UFAaHwuC0jWwxOIKVow8P4hc/fWnj7IsnkJksTnZeSX5JGZ5IrqlrpNAYsP7lFbWx99IhvgBHIBlbuzp4BMLAjr6b2t7Vx2Jzmtu6WGx5npYIEA4QLp8A24VuroxXIT2Ekyl0baNbR86qXdUwnppFcXmQT9DtI2fV3HzD2BxuY2vnGSVtNR3z04qaDCaroKT8m9PKhpbOlg5eXB4UGZd8/MINbSNb+LWlMMLfNXcoXNE4rajV3vWey4P8w2KPnFVz8gphfz8zPDkj75CC8uEzqofPqE5Oz/mHxh47fx2euCSMcIgvGBmfir6bWt/YZvRxdPf5q9qTlzXm0VhxhEN8gXfQ7eInLyC+4E5i+pFzapoGNmgsgcFk27n5Q3xB8ZOXrj5hEs6O3qNcx+KJ355RgVE3OT13VcP47FWd6jcNPIjf1tl76rLGpWv6o+NTXb39xy9c1zCw1jd3hN+FO3uHfnNa+VXNWy4PCgiP/faMin9oTPidpMGRiai4lNl5tMIVDSUNY9Nb7kOjEyG3E2GEQ3xBfHKmlaO3sFwPCkrhOxt/P2v1YcvJh85gXdO1OKSgfOScWmp2PoXGsHbyOXxWNTQ6kcP97sOEzp5+VW0ziC/40OG+5eLnGRgFl2lm6z41Mz87j/nmtPI8GitckZzFAcIBwgHC5VMB6SEc9pgkCu3/b++8n6LI+kb/Vt2qW/fHW+8/cO9Pb9X95d53d593n7i7zwb3WXV3jaw5oBIliIBIUhBQETCjIohZDCAgAhJFgiJByWkIM8Dk0DkS5gq9O44ziMAw2tN8p7qo0z3dPed8zvD9zDl9TjfNvH0WNU68nWBG0YzeYFqxfieGT13ptDhYiJ6zTNZiWM56cpf1OWeMvDPOamNYLiHpfHzSueCI+MRTUxdKhQzPeAahX9cyIp2cHsfMcvyJUxdzHk0N8pr92Ped09W3Ww/nfjM+3FIpNMNa90szLEdZTfOzr1mG5S3HWph0dPX9tt03v3BqIKT1gpPUB2e12TSpERRft8Wro6vX+jyv27p27Q3FCZLlZvh06z0lkAaFg8KlKTCRN5E/QvacqvAPxj6KfQvGMQAAIABJREFUZrt6++3D9wcPXJQdCJJuft3e3Tcwx9bzkGLUOqs0w7Z19VprbFFyBScRCIwo1b3TF8UXBYhOb7JcarGcX2eAeeHmf7N5CPnExMTY2BjLsiotTCrr/gghGD4CCDhC4NMqfFGiM5wECACB2QlAKxxa4dAKlyYBUPjssQ/eBQISIAAKB4VLU2COtF+lcSwoXAIBGooABGYnAAoHhYPCpUkAFD577FvwuxhBGUyYzoiIedEbURSfGs8Fy+wEUJzUG1ExV6XOiBhMGGY1XNS6RKBwULg0BSaNlrQjpQCFW0e6xUqbUJyk6ImJCZuhQmJbnZicpGnWaMKsB5AvFgRpnGf6wSrYm+mLE5OTYqs+m/xMTEwQJG2ZRm/NHxQOCgeFS5MAKNw60i1KGsNJjuNtwquYV8fHJ5A/7mC6KASkdBIEI8bHxf5TzPrbxXK8/aQ7UDgoXJoCc6T9Ko1jQeGL7hutwTQp+habddA3m80zNt0WnYwrnhDBCBtWIl+dnJzU6n9/GKsFOCgcFA4KlyYBULglzC1WQq01iDzK22cPWuHvq32XU7jZbLaf0Q0KB4VLU2DSaEk7UgpQ+Pti94K3g8IXjE6EB4LC3xnrCM8LdyTawrFAYNEJfASFjxi5k2X0igvE5/H4LxeJc5W0Cnl7y1URRm0HswQKfx9AguYftTCbM8kvEvCvU/CAe+QrBcuIezw8KBwUDu1XICBeAs5W+IsBdlkK/lk0Zr2sOEO0jvz+CIr3hXvX3T6jwhmGzS8stfRgNzS11NU3j4+PC1sam1tVGq3l3Y+f+Agd6SqE+y2D+NPhd74Jf4/HksrfPAHnHU2IqurtFc5xfOWzuuERlVBNHMeXV9XKFSPCantnT0NTS0NTi0o9VaEsx5VVVnf39gvvMixbUvGsVzbo1CpeeEf6xMSEUqmUyWQkRdqf5c2zdaEVvuitKDghEHCEgFMVbiD4jWmEtbwt6a0ZBEqKN3A7YhF7hRtMyKH4lH8udzObzQRJrd/qfeZiZnRccsrZywzDbtkdcPJcemBYzMNHT5wa2Wc5+UdQeEYNbal968S3ifiwQby9MjYKV2m0y37dcvnqHQ//sEH5sMGIuG3zuXTllu/+KNnAEM+PfbdyQ9yJsydOX6p90Tgyqlq53v3Kjbu79x4oKa9SjCh/+W1X5s177j4hZZXVs1SHg2/Zy3eu18KvXLnyzTffrF+/3tPTc0gxav9vAAp3JNrCsUBg0Qk4VeEFrYx1sLZJ57zUKUY1klmUGoPwyBN7hW/ZExgRkygoXKXWZt68ZzabSYpy9w5WqjRf/Ws9TpCdPX1xiWcdjN0LPnwuCqcZVqU1Lri+fj6N2HwBLKvetxd+2gXnZ/YD1Tqj8EgbG4UrRpR19U3j4+NpmbcLissfP6k4lJBiNpvr6pvCoo8OyYd//m0XgmI0zZjN5ural4+Kysxmc/nTWg+/sMpnzwtLKs1mc3HZU9+gyAVX1gcPXLjCfX19GYZhWfbq1avVdfWg8EUPuHBCILC4BJyq8AtVlCVM2yfSKvWzh1GXe3dUrWc53l7hGIZ39/YLChfi7/j4xPGTF9Iyb4+NjYVGJazetOdfq7c2vWr7YHR20g5zUbhGb3KkRv50BLX/Dghb/paEOHJmJx2rNSAsx9so3Gw2cxwXFZe0etMeE4JmXMu6efeh2Wzu6Op12+pd9rTm65/cEk9d2LI7oKauQagsnh/zDYq0VC7Pj3kFhDe3tDupKh0akZ6ZmQlPKnteygx309rLBTXN8KQy8V4DXlwRuu7ZnKrwi8+WmsJ1MyrcbDZbK5zj+JPn0wNCD5EULRsYWr52x5B8pLSiOij8iPPC+uxnnpPCdQ61lf8r3tUUPj272l7hk5OTDMteyrx9KCEl/eqdG1k5ZrO5vbNnh1cQRdNa3dSUwqfVL77/eZPZbDaaEJ994Zeu3BT46w1GD/8DGdeyZq8OB99deCv80qVLoHBQuOv6bAnm3KkKf9I5W0d6fqPEOtL1JMV8UOE8z+/dH5l8Nm1sbGosW9OrtojYRLPZjOOE2zYfB2P3gg+fi8IpmlFpDQtu8q4+996O9KD7Dv04WHCWZjlQpTVS9NSISxuFF5c+TTmXbjab6xtfrdnsUVz21C8k2mw2V1TVnjh18VFhaerlG2azubL6+Yr1O3CCWLfVq7SiWrjVLo4T67Z6Vj577uw77y5c4Xv27JHL5QaDISkpqe5FA3SkL0ElQJFdi4BTFY6S/Ob0mYezuV8nMGqpDGezboW/bu344uuVAaGHgyPj40+cNSGoh3/YoYSTPkGRt+/nLdjBDh44F4Xbx/N5bbn+fObhbN8n4UqTywxnMxhNm9z9Yo+d3uUbUt/wiiBI730RcSfOBh2MNRgRjVbv4R8Wn3Rul29oe2fP+bTrf/thTdDBI8GR8SfPp586n/H3H9cFhU+tnr2Y6WCVzXL4whXe3Ny8cuXKH374IS0tTame6lOyWWA4m2vFd8it5Ak4VeEsx4+YuG9P2E4q+z4ZRyTq7/e1wmcJuGazGcVwfmxs9n2c+u5HUDjD8WvTiS/enVT251js9FPaRhOiWrVphZvN5snJSZ3BaN2SRjHMunZMCPpp77C7cIWbzeaxsTGaplmWtT8LTCqTvA+ggC5HwNkKZzl+UM8dL6aWnyc+j8NXpBLJJZRCxJOIHPeH/XA26/guzvRHUDjL8TjFP2hkNmYQX8TjXyXjPnfIFwOsmCeF23eki7P6bHJlL9+5Tiozm81zvBZO9asbq1ipLqMyBoazuZzMlmaGP4LCHZeia50BFO5a9TV7bu1b4Ta+FOHqx1A4VtuBtg1LdcHbFZqL+TAifWlK0bVKDQqfPYIv4F1Q+AKgifYQUPg7l8Mt18JN+dVo2QvpLvWaC3mgcNeS2dLMLSh80eWh1bviw0aJRecgjRO6nMI/0sNG6e5+5dGVUl2wypvQkb40jehypQaFL7ppUJzk+E85Nm2+XbsTExMf51r4oqP+CCdEMMJ65Np82X78/Tmex3DShsziXwunu/tGor+R6oKWXgGFu5zMlmaGQeE2wc7xVYblERQnKWZiYvLjR/B5feLExCTNsEYEY1jxTutyvEYcOQPNckYEoxnWFWpzgiAZBMXtRwg6S+HP/f988tf/uLv1/w5Ffi0lnYPCl6YOXbHUoHBH4vssx2IEpTOiWr1JzIvOgED7e5ZKtLyFYITOgIi5KrV6k86IYgRlybN1wikKV0R9/e///b+9DPhL8i//Ub33S1C4KwoA8uzqBEDh1pEO0kBAkgScovCGwL/89X//z2rfL/d9/b/Kvf8ECnd1GUD+XZEAKFySIRsKBQSsCThF4Yqob/7j3//H/W3/79ya//Nw53+Cwl1RAJBnVycACreOdJAGApIk4BSFj0R/0x7yt4c7/rNoz+f94V+Bwl1dBpB/VyQACpdkyIZCAQFrAs5SuJS0bV0WGM7mijJbmnkGhVtHOkgDAUkSAIXPb/4bKHxp6tAVSw0Kl2TIhkIBAWsCoHBQeI4r+gny/EECoHDrSAdpICBJAqBwUDgoXJoEnKfw8qq6otKnwvKi4fWi3DyktaPnQV5RcVlVR3ffHE+YnV8kG5TX1jc9rX5hE50vX8saGVUfPHS8u2/A5q1ZVo0Ieiwl1YRiKrWOYbm+AXlq+g2KZmc5RAJvMSzX0zeQnV9UUlGj1ujtS4RiuFZnYFgu5dzlV22d9juwHF/57Hnls+czvjXjRsWI8n5eYVHp06aWdoJatGeSGowIguIzfuIsGwuKK+TDytv38958Z4TdhhSjuQUlsxwinrdA4aBwaQrsg41Uye/gPIU/q2vILyr76/ercx49aXrdzrAczbzjOetVkmKs4531WwzLW1YTks4FR8Q/flK5dotXwZMKlpt6y9rlDMsJqzTz++3GjianNr1uv/vw8Y2sh8JHCGdjWM57X4RsQD4wNCzckNJGw5YPtSSEwzNv3k8+e5nl+NDoBATDKZrd5RPSIxu0zr/00s2tHSvW7Swqrbp1P3fDDl8U+12BFjgVz54nnUljOV4xorIIkqKnqtVSQXqjSW9E7OEIu1m2W74MhSWVngHh5VV1u3xDEk9fmj7V2y+D9arlI4STCLVv85WzZPX6nZwn5c+EPS1fGJtVy87CdqVat3VPIIoT4TGJPX/84GNYzsM/DCds72YqHCKqv6BwUDgoXJoEnKdwluN1BtPXP7kp1VoTgoVEJQSEHvYNilSMKJPOpkUdSdrjFxYRe4Kg6Njjpw9EH/UKDFdpdC+bWtx9Q3yCok6cuqTVG1es37kvLPb+w0IhICYknbt+J5vl+BOnL8UeO/Ugr9DDP8zDPyzvcWlXT/+K9TsDw2JfNrWEHTrmExS5ZrNnb/+QtcK7e/t37w0JDIs5lpIqKLxXNrR+m/frtq6LGTf9Qg55BYa3tHc9LHiy3XOff+jhTe5+EbGJ67f5FJU+tUTk/RFxvbLB0orqb1duDI6IU2v1125n5xWWERS972DsvBr0lnOKPxF3/EzFHw3owLDYwpLK4Mg4r8Bwn6DIwAMx8mGlZ0D4z27uxaVV+8Jia+ubfPdH+ocecvcNCY1KCI6I+/6XTZ09spxHxXmFpaWVNQeij25099vk7qfR6sMOHQsMiw2NOqozmDbv8vcKDI85dkoAUlhSeSjhJMvxj59UbNkT2NDUsss3xCsw4uzFq2qtftmvW/aFxTzILUo5l+4VGL5hh2/N88YHeUV+IdF7/A4EhsUeiD66erNHQ3PrqEqze29oYFjMvoOxvbIht+0+m3b5v27rzM4r9AuO9gmKeFb7sq2z5/tfNvmHHMp7XLZx597AsNjwmETsDz3n5BcHR8azHB8ek9jR1Xc0+fzFK7dohj2SeLawZOq7EX/iHEHOfGc0MVQuKBwULk2BSb6R/cECfhyFv2h49eOqrV29/Zcyb99/WHj85IUXDa8Ikv7nig3Do6pffnOved6IYgTDcuu3+VQ+e86w3C7f0NaO7mWrtgwpRi1BMCHpnE9QZMaNu9//srniWR1FMxqdofZlo3/o4c7uvi//+Ss13dAnSEoxonTb7nMvp8Ba4THHTqecS+/q7d+yJ4BmWO99Eb2ywV837mlu6fhx9dbMm/eMCMqw3IPcwrBDx1iO3+EVpNbqC0sqffaFW/Kww2u/kPYLiTJN98fW1jedS7vOcnxTS4cl6Fv2l0Zip3ewpfWZfPbyqfMZ/iGH0q9lMSy3Lyz2SfmzksrquMSzDMvtDY5+Vtewe2/oo6Iyrd4o4IqMPZGdX/Qgt/DhoydC38lO7+D+Afmj4nLf4Oiunv7gyIS8x6Xrt/lU1dRbiBWWVG7ZE3jnQf5KN/dzl67SDGcwIi3tXT+t2abW6r9Z/ptiRMlyPEnRSpX29IXMlHPp9x4+vnUvT6XR/fm7VYoRVdnT2is37uU8Kg6PSezq7fcPiW541ZZ+LevxkwqaYb/+ya2uvjnvcckXX61sae9at9Ubw8nquoZvV25o7+ojrbruT55PT718Q1D4/vDY89NpluPTMu+cvXiV5fi+/iGbngBLKcSQAIWDwkHh0iTwcRT+rO7lhp178wpL8wpLO3tkx09e6J3uef5lw275sLKrR3Yj6+HWPYHPXzav3epVW9/EsJxnQHh9Y8uyVVv0BpMlCMYnnYs8klRc/qyts5ekGK/A8KQzaTfvPgyYVviKdTtZjldpdLt8Q7OyH+30Cb6b/Y7Co+JS4pPOCdnAcMJa4aMqzYO8It/9Udl5U6Y5npL65lSeAQc1OkNJebW1wt19QoRg/VbhLxqF+G7Jp/QSJ05fyi8sFXrFvfdFVDyr8w85dPPu1LWJ0KiE3IKSksrqI4lnrBVeVlmj1Rt9gyJZjj9y/IxF4ThJBR2Mvf/wMcvxd3MKdvmGWr4Y67f5NL9ut9ArLKnc5RtaXFbV0NxKM2zy2cuRR5Ie5BWtXL9TULjBiGA4scfvwLXb2UcSzySfvXzv4ePcgicaneEv361WaXRVtfXp17OysgvCDh0TPmVEqU6/llVQXI7i5Nc/ud15kJ9XWPqoqPxVa+dGdz+SokmKqXneeObi1W0e+yzfvVOpGecvT/1KC49J3Bsc7bs/Cp++G3na1TtnL00pXOQLKBwULk2BfbCRKvkdPo7CR5SaXzfsvnL9XlD4kWe19dYKlw3Ip1ts5V6B4Y3NbW+GC+3eG5px4+4evwMand5G4ZaOdJbjCYre5O6fdCbtSOKZgNCYzu4+QeGjKs2GHT637uV+t3JjTn6x0Aq/N30tvOxp7YYdvrfv53vvi7B0pP+6cU9Dc+t2z6C0q7ePJp/PyS+2VXhFtc++CEuM3h8eJ/SWxxw7febiVaMJvXb7Qe7jEpKit+wJ6OyWWfaUUkI2IF+z2ePsxavxJ87uDY6iaMY/5NDWPYEZ1++6bfMZHlV39fRvcvdvbunYGxxdPd0K/13h+6PeUXjBk0dF5f9cvuFudkFBccXg0PB2z6B7OQU+QZGNr9rsFS50pAsko+NTwmOOp16+/tPa7RaFIxi+fpvPjayc7Z5BMyn8Zcb1u+2dvcvXbr+R9dDDP6yvfyi3oGS7Z9DA0HDssdNJZ9KSzqSFRMa3tHcLCq+tb9rqEfggr2iH136LwrPzi4LC4xh2SuFvqvjg4ePHUlJpho2b6kivZDk+IiYRh45067ujuHQa5oVL3nySKaBTFU4z7JBiRBgmpjOYOnv65cOjNMNqdQahl1IxohQ6w1vau4ZHVcLYoul2eb9p+gmPwv4WF2p0BoPp7XgogxHp6u1XqrUqjY4gafnwVLcqw3KKEVV3b/+oauoavEZnwAnKiKAG01Qn+YhS3d7Vq9EZWI4fVWlJilaMKHGCMiFYZ7dsYGj4jZyMCKb9fQcNzbAIhitVWksebmTlCOO2TAjWKxskaWbLnsAe2QDDcr2yIZcY3GQpy7wSGp2hpb1LNihHMYLleP+QQ/dyCto6ewSYb647DClGtDqjUq3FcHJUpUExnGZYpXoKnUZnMCGYEcFMKKYzmPqHhvuHhqcrl9Pqja3t3SNK9ZurJ8OjKqF1K2QMQXG19u3od4wgu3sHhkdVw6Pq6a/W1HeJ5Xi1Rt/RLVNpdFq90YigwjdnUD5CMyyGEwYjwrBTfTMd3X2jKg3DcgRJyQbkJhQjSKpvYKhHNkiQFEFSw6NqYeykfFjZ2S2zlIvleK3OsG6rN4oTaq2epGgMJxUjU19XD/+DQoblw0roSJ9fS1fMjgeFS8Zwki+IUxU+L0O4ys56I3I0JVUYxD7trdFTqVdsxlS7SlkcyefVW/ebXrc5cgbXOja/qEyl0VnnWaXRPcj9faCl9XYRpqEjfX4/L0DhkjefZAoIChdhwIUsAYHFJTBXhRsMhomJibGxMZZlVdqpriqbRS6XK5VK0+MXdHefmJvRDuYNFC4Zw0m+IKBwmxgFq0BAegTmqnCZTDY4OBgaGurn51dT93ZugIUIKFzySoACuhYBULglOkECCEiVwFwVPjk56ebm1t7ejiCIbEBujwMU7lrxHXIreQKgcPswBVuAgMQIzFXho6Ojy5cvHx0dpWl6RKmxpwAKl7wSoICuRQAUbh+mHN9CkDSCEihOin9BUMJ6ELjjZZfeGTCCQjCXqU2CnOFm8nNVuMFguHDhwunTp6uqqp7VzHA7e1C4a8V3yK3kCYDCF105FM2iGGF2nRdOUmDx930NMGJqvpnrVKYZxaaeCGNTnLkq3Gw2j42NNTY2VlRUyAahI71b8gKAAro6AVC4TbBzfNVgQicmJlwo6JvNZuFOsY6XXXpnQFzq15jZbB6fmDCYUJuKmIfCYUT6SPQ3MCLd1cW2dPIPCrcJdo6vqnVG1/K32WxG0Km7tcBiT8DlFG42m9U6o01BQOEwLxxusCpNAqBwm2Dn+Kpaa3A9hU/fcM3xskvvDK6ocPsZ3aBwULg0BbZ0WtvvK6k0FK5EuHvNTEoZfb6KLupgMOpTNihB4Q6KfEDH3WpgkkvptFq6RmZ7WdfBk8/3cFD4O/9LMJztfZEUtgOBT0LA1RVOMXzjELssCf8sGrMse24QCgM332C9WPvPqHCaZnh+TLStc0QcrXCC5h++Zj479LYqP4vGDuZRGvST1aa9wicnJ0mSGh8fF21tQit8fm1u+1u5wbXwT2Ij+NAFEHB1hT8fYP8U+07EF0T+2yWCoN9pPyyWoT94HnuFp2Xe3uG1f/OugNb2LnHGfZEo/F4z80XMDLW5/z75QexO2sFG4ZOTk4cTUjz8w7z3RWC4SOcdgMJB4dBtvlQIOFvhOEmjGIlghJOWvTffaX9bGuKfRWO36p34ufbFwQhKeFaVjcJNCPrjqq1GE9LS1nn46EnXVTjDcvj0DGn7si/KFoWOWJ7y3tos6/yotYn/UZs2Cu/p6/925UaaZq7eun/rXq44axMUDgpfKgJbQLNVYoc4VeEoTipGNU5d/nEctda2dXplqsmpH21/cq3eNPXsy3eHszEsu2G779mLmf4hh67deiDOoD+XVrjBhNkXeRG31HZqvzwyQxNcqNOge8ZF/Ky5nMqIYCzH2yjcaEL+udztzv28zbsC2jt7xFmboHBQOCh8qRBwqsKNiHODvmJU81nsexX+2VF0LpF6EfcZVk49G9tG4SiG7/QOTj57OTwm8cqNu+IM+nNRuEprWERW9qeqatN+dvi9Ct965WP/INNMT82yUXj/oHztFq8L6TcCD8SUV9WKszYXrnAUnbqnAUVRvb29/YMK+4sTMJxNYm04KI6rE3CqwnGCGlZq7YP1Im5ZloJYt7yt0xsyPmrQH1Zq9capW2rYKLy7t39fWIzZbObHxjbv8hdn0J+Lwk0o7tTarO/W/jX+vT/IInKd+wPC5js5rNQKc+VtFH79TvbxlFSz2dzS1rl2i6c4a3PhCg8ODmZZNjU1NSUlpaOrFxTu6vEd8i95Ak5VOMvxFM2SFENQtJOWyIektbbfpg9jT7ud9aEzloWkGSHi2Sh8ZFS1Yv3Ojq7ex8Xle/dHiTPoz0Xh07XpxKrUo/Tqc++5Fh6DtQ478aPtK5T6ozZtFF75rO7XDbuH5COnUjNij50SZ20uXOFxcXEYhkVFRcHzwrWXC2qa4QarS6U72nVN72yF2/+OX9wtvRruL0dniPteN0mKEcWI9MnJyZrnDdHxKXEnzo4o1eIM+nNU+OLWnf3Zng+yX8bN0JeeUjbDozvsD3fGFhuFT0xM5D8ujTySdC7tGsOw4qzNhSs8KSkJw7CDBw8aDAaVRm8PFDrSXTfWQ84lScDVFc5yvArhVp7GP/9jMvHnh7CAe+SnmlFm35EuzihvkyuRKJzl+OcD7BeHMUttfnEYO1pCUeyn+TVmP5zNhps4Vxeu8OTkZAzDWttaExMTXza+AoVLMuhDoaREQAIKZzlej3PVMjb3NfO4jWmQs5/Q36Bw+7A/3y1KhKvsZR++Yoo7mQ7lJ7upi5Btm1a4OJ1tk6uFK9xsNk9MTOTl5VVUVAwMDdvXHLTCpRT9oSwSICANhduHmk+4xeZauE14FeeqeFrhn7DiZvzopahwo9H46NGjlrZOeyKgcAkEfSiClAiAwu3DlINb1DrjpDhF/f5cwZPK3lfpLqfwyUl4Ulk0zAuHYWhLhQAo/H2xe8HbjQg2Pu5izwtHUHzB5ZX2gQgq0ruovu/32Pj4hHBTGut6gSeVzU/qcI90KbVTpV0WULh1pFuUNM2wCIq/uZn2+4KsqLZPTk6iGEmQn2zI96Iwd95JcJLCcNKFatOE4jRj+3g3UDgofKm0SqUtbPvSgcKdEf0pmjGhuN6Iin8xIjhJgb9nG/FOUrSr1KYJxS0z2q2/2KBwUDgoXJoEQOHWkQ7SQECSBEDhoHBpCsy+VbrUtoDCJRmyoVBAwJoAKBwUDgqXJgFQuHWkgzQQkCSBuSpcLpf7+vq6ubk1NDaotHB3NrjBqjS1J6WWOihckiEbCgUErAnMVeHx8fEymUyhUFy7dm1EpbE+hZCGeeFSiv5QFgkQAIXbhynYAgQkRmCuCg8KCqIoamxszISY4B7p8JgTCRhO8kUAhUssWENxgIA9gbkqPDk5uby8vLa2dv369SNKaIVDRzp0pIudACjcPt7BFiAgMQJzVTiGYXfv3r1y5YpCobC/0zrL8dCRLvlWHRTQtQg4T+FNr9tijp0+cvzMlZv3FCMq+5j4ouG1/UZhS1b2o6i45LjEs/dzCym7+1TMeNT93MLW9p7rWTmV1S+sd8AIMiQqQT6s/G2H76B8hgc3WO9sne6VDSUknbPMss19XHIjK6e1vftG1kPr3ZZIurisKjrhZFzi2et3snGSWkCpG1+14sRCDlzAZ8EhNgTmqnDhMSdjY2PwvHB4XrhrmWzJ5tZ5Cn9Y8CT2+OnuvoGjyee37AkkSBrDyUH5yJBilKRojc6waZff8KiKZliVRtcrG9IbTJa4sz/8yLXb2a9aO/+1ZltBcQVBUgNDw/JhJUUzGEGOKNVDilEMJzQ6g2xArtLoGJZLTb/xouH1iTNpuY9LGZYbVWkGhoZRnEAxfM1mT8WIcvpzGRTDZQNyoY9Qb0TUWn3/oFyrNw6PqgflIxZhMywfEhmfW1DCcrxmah/FxYybSWfSWI738A8T7n5lQjBLhiWfOJWakV9Y1tLWtdLNvaC4wmBCtDqD8MtMqdYODCkEPRsRVDagGFVpGZYjKWZ4VCUfHiVIyoTiu/0OtLZ3kxSt0uj6+ocMRmTqUZ7vuRWJ5Hl+5AKCwmFSmdg7hJesgx0suFMVfvJ8BsvxNMN674tQafRHU1KPpVzwDYq8eOVWdV3D8rXbc/KL2zp7d3oHp1/L2uG1n2F/f7Lk/vAjpZU1LMf7hxy6cuNu5JETCcnnD0QffZBXVPui6at/rT9+8kJDc8u6rV7n067/7ObeNyC3VnhBcbmHf9jFK7cOHj4uKHxgcPhP3/wo6JvuAAASL0lEQVTS2z/kH3oo/XpWwIGY9q7ec5euum33STmX/uPqradSMzbs3Psgt1CIrQRJbXT3U2v0g/LhjTv9zqVdW73JQ1B4aNTR9s5end64yzd0Ye3Rjxy+F+XjTqVm1NU3sRx/NPn8jayH6dfvum33uXU3t+LZ8/3hccnnLgceiMFwYu0Wr8tX7+zdHzWkGE2/lhUalXA44eTho6c6e2SrN3lk3Lhb3/ja3Sf49v28oIOxFM0EhsUWlT5dlBzCSWYhAAoHhYPCpUngIyic5fiI2BO9ssEe2eCDvKKI2BP7wmIZltvpvZ/leJwgy57W3rqX+6/V23R6oxCGgg4e8QoIjzyStGmXf69ssFc2mJ1ffCTxTGRsUs3zxmW/bmFYjiDpZ3Uv7zzI//6XTdV1DRaFP3xUssM7+ODh49duZy9btUWnN67Z7DkwqPj8Hyt6ZYM+QZHHTl4oe1pL0czpC1fOXMhkOX6XTwjL8Tfu5MQeOyVkAMPJNZs8WI5Pv551Lu3atLpSBYUnJJ0vqaieuiw4rLT85pglekrjrVOpGd77IqLikldv8ujo7rt8LUtQ75rNntHxKWmZt//2w5ruvoGN7n5JZ9Kqal+SNOPuE5J5637mrQerN3mgOLE3JFqt0bV19mzeHXAjK6e9s5fleMWICsNJaSAScylA4aBwaQrMwSasBA7/CArX6Aw/u7mbEGy7Z1BbR09RWVWglcKv38kJjog3odg2zyCt3iDEwf3hRwpLKlGcICkaxfA/f7eqtr7pafULQeHrtnixHF/f+HqH136lWht4IOZ5wyuLwnMLStx9QkoraxAMn1rQqY50i8IpmhlSjASEHr59L+/0hSuXr95mOd4rMJzl+Nv3898qnCDXbPZkWO76nRyhLyH22GlB4bHHzzyreynmeO2MvJ1Kzah89hzFCWL6huqXr2UJEHbvDX1W91JAPd15TvfIBqc21tb7BEWaEAzBcAwnWI7fGxKtVGtZjkcxvOZ5w7JfN9s/jcMZOYdzvunNAoWDwkHh0iTgPIXnPipZuX6n7/4or4CD2XlFDMuFRh/dHxG3NzhKUHho1NHo+JSyytp1W70OHj4utJiFgGvpSGc5/s3VU8+Ag75TR8VExSXXPG8UFN4jG/x1w66I2BOrN3q8aHh1wepa+IuGKbsL50emr4UPDA5//tWKrt7+g4cTQ6ISvALDO7r6LAr3/kPhR46fFjJATTciO7r71Fq9u2/IvoOxm3cHCAr3C45WqbUmBNu1N5RY0MAuV5TKqfNXhI50IfMWhb9q7fQKDA8KPxIalaDRGbwCw0OiEjwDDg4qRvILyzz8w/YGRyedSaMY9uylq54B4VW19Tu8g6Pikg9EH6Vo1j/0cEVVnSsCca08g8JB4dIUmASa0Q4WwXkKnzHG2TyXWhg+RlL0BxtkMw5mnu5On3mQM81yJnTm4WYmFJtLB3j6tayT5zMYlmNYHsWm2pHC8Cu/kGghTdG2j3ScsciS38iwnHW1ItjbR4/bVJBQ3TTDWkYCWsYPSp7Spy0gKBwUDgqXJoGPrPBPG8jm9ekYTpZW1thMaevuG+jo6pvXeWBnIPDJCcxV4ZOTk6WlpaGhoZGRkXX1Dfb5hnnhDraZ4HAgsLgEQOH2YQq2AAGJEZirwhUKxXfffadSqWQyWU1dvT0FUPjixl84GxBwkAAo3D5MwRYgIDECc1V4d3f3t99+mz39am3vsqcACncw4MLhQGBxCYDC7cMUbAECEiMwV4UrlcoDBw5kZmbm5uZW1Ty3pwAKX9z4C2cDAg4SAIXbh6lF2UIzLEkxBEWLe2HmNShvulAiL9HSzd4sw0LnqnCe51mW7erqamlpkQ8r7f8TQOEOBlw4HAgsLgFQuH2YcnwLipEoRjIcz/FjYl6mb61DmZC3Y8hnKTuCEjoDakJwE0LAIkoCuM6AIH/MnrCuyrkqHO6RPhI9NXYdLb0C90hfXNPA2ZxEABRuHekWJU1QNEnRZtd50Sz3wVuk4SRtMKIYTsEicgIGI0qQtM03GRQOk8qkOaXKSV50odOCwm2CneOregMyMTnpOgafyqn1xO4ZCWh0JhQjRW4vyB6GUyhGanRvnxgk1KYzFD6A12RLdWH6XmvTHg31Dee09MICBMRMQDao1F4p1Gu0g4OD9rFbpTHQDM3z/MTExKSdliYmJoTHEmp0v98Y1f4MS3CLWmt0LX9PKXym3lfrulOqdSBIVyEwqtZb193i32AVLW/Wpj2CBQgAATEQMNyt0Gt1oHCbqLfgVbXWID2Fj6r1riIwyOeoWmfz7V3MVrhCoTBqtAalGhYgAAREQUClUY6OOlXhxZ2M7wNy1UXCP5ss75H4fUlB4SDRT0vAqQpXDMILCAAB8RGQy+U2v9xZjne8I31AxwVkkZ9FY9ZLaDY5bPz90eD2H+rqW2ZUOD821tHVK9rW+Qc70qEV/mmtPK9Pd6LCXf2fE/IPBJYUAQcVTrN8cI6tvwWXh+dTDDv14BDpLfYKHx8ff/Dw8T+Xu0lP4VU19S3tPRbBPH5S2devsKwuIGFE8BcNr+88eDQ8qhYORzEyt6D0QvrN85dvpF6+eSH95pPy6uz8JygGw+NnJrAICjeZTC1tndL754QSAYElRcBBhVf3sdaNb5t0VRei0Zsks+hNqPC8NXuFR8en7PYNlaTCU85nZOc/sag6NOpYdV2jZXUBictXs5LOpOXkP1m31csyhq6+sfVJefVv2/ceTb7wpLy6oamtqLRqASdfIocsXOE8z3t4eJhMJqPR+Lq1Y0kFOygsEJAeAQcVfrmGstG29eqVKr1iVCOlRa2dGplvr3D58Eh3b//SUXhHtywqLtkv+NChhJMqjf7O/Uep6Tf3hcWeuXD15PmM4Ij4+sYWowlLy7wTHBHvF3LoZVOrRa4vG1sRlGjv6luxbqd8WGXZjuGUd2BEdl4xhlMt7d0nTqcNj2hij50Oj0ncGxz9sqn12u3sgieVGE5V1bxMv3bX+kCppvsGFBnX7xlNGIZTnT390XEpGp0Rw6mFK7ykpOTLL788cOCATqd70dAkvYgGJQICS4qAgwo/WzmbwlPLpKbwYaV2RoWbzeYlpfD0a3dfNrWq1PodXvvr6l8dTU69fDVLNqD4/KuVpZW1VTX12zz2vW7rDgiNGVVqC4ortnvus7ZsZ0//X79fExF7woTg1tstCq990ezuE9I/OLx+m/fIqObew0KfoKievkG/kEMoRnoGhJdW1FofKNV0r0z+1+/XBEckKEbUy9ft+GnNdo3WMYXjOP7dd9/J5XKVSlVZVbOkgh0UFghIj4CDCs9roa2b3TbpB/U6KTXBR1Q6YVCYfSt8qSm85kXTgahjhxNO/ezmXl3XeDQ5tb2rD8OpZb9s6ekbam3v+eHnLUqVLiHpfGRskn/o4bWbPS2WFe5NO6RQ7vINbXzVbtlu3Qq3KDzoYByGUy8aX+/xC0NQ0jswsqdvcIfXfqFhan2sVNNFJVV/X7bu78vW7vDaL/jboVY4y7Lff/+9Xq8HhUsvmkOJliABBxWuxfi/HXtnLLrF4v9IxA24BMeyLcFW+N3sAp3epNObjCZMuBa+2ze0o6tPb0A37PR7VttwNDm1r1+O4dSPv27p65cLCr+X83ind7DRhD0qrli1YY9FsUEHjwzKR/UGxD/kcO2LZsv2GRUeHJmA4VR9Y4ug8NTLN/aHx12+mmV9lOTTtS+aN+zYq7KauL/wjvSxsbEffviho6MDFL4Ewz0UWXoEHFQ4y/EvBtj/OmJr8S/j8A7V0ppURpLUy6bX0huRfio1Y9mqrWu3eK3d4nXxym1B4ddv52zeHeAZEO4bFJVXWHY0+YIwTP13hXf0/Gv1ts7u/t+2+75pMfuHHv7zd6ssN2Cvqnm5bNVWt+2+B6KPCVd2LQ72DozImR46N9UK9w0dGBoJeVfhHV2yP3+76nVrl+WQpZBAMVJvQKxLunCFm83mysrKpKQkg8HQ9KpVehENSgQElhQBxxXOcHxFD7vpEvGPY/hnMdhXx/EtaUS1TMp3d5mxI1208hYytujzwrU6o/HdK9nWjhHSKEZq9Sb77SaEUGsM9ttn36IzIKUVNeu2elt+Dcy+v4TfdUjhltsmq6YHZy6peAeFBQISI+C4wgUgBM0bCV6PcUaCJ2hp9p9bqh4U/kns2NLW8+OqrT19Q5/k00X1oaBwiYcYS6yBBBCYncBiKXz2T5HYu6BwUflsCWYGFA4KBwJAYIoAKHwBPy80epP9U93E3pGOErOXFG6w6kI/BUDhEL6BABAAhS/wO2BC8fHxcZE72yZ7H3xeuFIDDxud+W6mIlS7UuO0h43O/kMP3gUCQEBUBKAVvoDqoFkOQXGeHxN/W3xycpIfG0dQgqSZ2UuqM6AoRopQV5AlGwLTA9RRm9pctIeN2pwXVoEAEBAzAVD4gmvHhOCjav2wUivmZUSl0xtRhv3wBD+KZrWGGUaP2/gDVj85AZ0BoRjbGR+g8AV2qS34/x8OBAJiIAAKF0MtiCQPBEXrjejUY2l0RljESEBv0htRgpqhQ2WuCkdRlOO4sbExlmVhUplI/vEgG0BgwQRA4QtGBwcCAfEQmKvC8/PzlUolKFw8NQc5AQKOEJijwrV6w1w6Yx3JCRwLBIDAggnMVeEDAwMMw6jV6ry8vLaO7gV/HhwIBICAGAjMUeEGI4ITlBgyDHkAAkDAnoDBhFI0xXHc+Pi4zSjLf7OenJCZmanX6yMiIpqbm4cUo/Yngi1AAAi4EIEPKnx8fJxlWROKGRHMhcoFWQUCS4qAzmCiaIrn+Tkp/M6dO/7+/nUvGpYUIygsEJAegTkqnCBJncEEfenS+wJAiSRAAMNJowmhaXquCpfL5SaTaVAxIoHCQxGAwFImMBeFcxxHURSCoiYUGuIwdQUIiIsAzbAanZEgCIZhxsbGPtAKv3r1ql6vv3379qpVq0rKny7l2AdlBwISIDC7wicnJycmJniepxmaIAiDwWgwgcXFFcEl8CWEIiyYAEUzaq0eRVGSIlmWHR8fn5iYmO1auNlsFp5URpKkUqNb8AfDgUAACIiBwFwULsxAISkSwzC9wagzmDCCIikG+tXFUIOQhyVIgGJYnKRQnNTqjQiCEATxvl50s9n8znA2i8JhXvgS/N5AkaVH4IMKn5ycHB8f53meYRiCIFAUNZlMWq1OrdWPqLTDoxpYgAAQ+KgElBqlRqfR6nV6PYIgOI5TNMWy7NjYmH0THBQOnWZAQMoEZle42WwW+tLHx8c5jmMYhiSn2uKCyE0mI7yAABD4+ARMJhOCICiKTvmb+t3fwlVwm150ULiUw7f02pRQovkS+KDCrS0utMUpiiIIAsdx7I8XCi8gAAScT+CPfzgMx3GCIEiKpGmaZVlhIPqMTXBQOCgcCEiZwFwUbmNxoTlO0zRFU28XCl5AAAg4k4DVvxs9/WJZVrjf+Yyj2Mx/vOBauJQj+HwbbbC/xAjMUeHWFh+bfvE8z8ELCACBT0GAn34JU8gmpl/2/ed/GByGs3GgcCAgWQJzV7hgceHS+MTExDi8gAAQ+HQELOaenH5ZhG2fgFa4ZMO3xBqUUJwFEJiXwi3RQYga8BcIAIFPS8DyLzlLwlbhs+wKbwEBIAAEgAAQAALiIQAKF09dQE6AABAAAkAACMyDACh8HrBgVyAABIAAEAAC4iEAChdPXUBOgAAQAAJAAAjMgwAofB6wYFcgAASAABAAAuIhAAoXT11AToAAEAACQAAIzIMAKHwesGBXIAAEgAAQAALiIQAKF09dQE6AABAAAkAACMyDACh8HrBgVyAABIAAEAAC4iEAChdPXUBOgAAQAAJAAAjMgwAofB6wYFcgAASAABAAAuIhAAoXT11AToAAEAACQAAIzIMAKHwesGBXIAAEgAAQAALiIQAKF09dQE6AABAAAkAACMyDACh8HrBgVyAABIAAEAAC4iEAChdPXUBOgAAQAAJAAAjMgwAofB6wYFcgAASAABAAAuIhAAoXT11AToAAEAACQAAIzIMAKHwesGBXIAAEgAAQAALiIQAKF09dQE6AABAAAkAACMyDACh8HrBgVyAABIAAEAAC4iEAChdPXUBOgAAQAAJAAAjMgwAofB6wYFcgAASAABAAAuIhAAoXT11AToAAEAACQAAIzIMAKHwesGBXIAAEgAAQAALiIQAKF09dQE6AABAAAkAACMyDwP8HwIR/UhU7CwwAAAAASUVORK5CYII=" alt="" />

(一)单GPU训练的三个基本步骤

当你在单个GPU上训练一个模型时,整个过程通常可以分为三个步骤:

  1. 前向传播(Forward Pass):将输入数据通过模型,得到模型的输出结果。这就好比把食材放进烤箱,等待美味的蛋糕出炉。
  2. 反向传播(Backward Pass):计算梯度,也就是找出模型需要改进的方向。这一步就像是检查蛋糕的口感,看看哪里需要调整。
  3. 优化步骤(Optimization):用计算出的梯度来更新模型的参数,让模型变得更好。这就好比根据反馈调整烤箱的温度和时间。

(二)关键参数:批次大小(Batch Size)

批次大小(Batch Size)是训练过程中一个非常重要的参数。它决定了每次训练时输入模型的数据量。批次大小的选择对模型的训练效果和效率有很大影响。

  • 小批次大小:在训练初期,小批次大小可以帮助模型快速找到一个好的学习方向,就像在迷宫中快速试探出一条路。但如果一直使用小批次,模型的梯度会比较“嘈杂”,最终可能无法达到最优性能。
  • 大批量大小:大批量大小可以提供更准确的梯度估计,但同时也会让模型对每个训练样本的利用效率降低,导致训练速度变慢,甚至浪费计算资源。

(三)批次大小与训练时间

批次大小还会影响训练一个给定数据集所需的时间。小批次大小需要更多的优化步骤来处理相同数量的数据,而每个优化步骤都需要计算时间,因此总训练时间会更长。不过,只要批次大小在最优值附近,模型的最终性能通常不会受到太大影响。

在大型语言模型(LLM)的预训练领域,批次大小通常用“token”(标记)数量而不是样本数量来表示。这样可以让训练数据与输入序列长度无关,更加通用。

(四)挑战:内存不足怎么办?

当我们尝试将训练扩展到更大的批次大小时,第一个挑战就来了——GPU内存不足。当我们的GPU无法容纳目标批次大小时,该怎么办呢?

首先,我们需要理解为什么会出现内存不足的问题。在训练神经网络时,我们需要在内存中存储以下内容:

  • 模型权重(Weights)
  • 模型梯度(Gradients)
  • 优化器状态(Optimizer States)
  • 用于计算梯度的激活值(Activations)

(五)训练过程中的内存动态变化

使用PyTorch的分析工具,我们可以清楚地看到,内存的使用并不是一成不变的,而是在训练过程中不断变化。例如,在一个训练步骤中,内存的使用会随着前向传播、反向传播和优化步骤而起伏。

  • 前向传播:当我们开始训练时,模型的激活值会迅速增加,因为模型需要计算输入数据的输出结果。
  • 反向传播:随着梯度的计算,内存中的梯度值会逐渐积累,而用于计算梯度的激活值则会被逐步清理。
  • 优化步骤:最后,在更新模型参数时,我们需要使用所有梯度,并更新优化器的状态。

有趣的是,第一次训练步骤与其他步骤看起来很不一样。这是因为PyTorch的内存分配器在第一步中做了很多准备工作,为后续步骤预留内存,从而避免在后续步骤中频繁搜索空闲内存块。这种机制虽然优化了训练效率,但也可能导致一个常见问题:第一次训练步骤成功了,但后续步骤却因为内存不足而失败。这正是因为优化器状态在第一次步骤后开始占用更多内存。

(六)如何估算模型的内存需求?

为了更好地管理内存,我们需要了解模型的各个部分分别需要多少内存。首先,我们来看看模型的权重、梯度和优化器状态。

对于一个简单的Transformer模型,其参数数量可以通过以下公式估算:

其中,h是隐藏层维度,v是词汇表大小,L是模型层数。从这个公式可以看出,当隐藏层维度增大时,h²项会迅速增长,成为内存占用的主要部分。

接下来,我们来计算这些参数的内存需求。在传统的全精度(FP32)训练中,每个参数和梯度需要4字节,而优化器(如Adam)还需要额外存储动量和方差,每个参数再占用8字节。

如果使用混合精度训练(如BF16),虽然计算速度更快,但内存需求会略有变化。在混合精度训练中,我们通常使用BF16(2字节)进行计算,同时保留一份FP32(4字节)的模型权重副本,以确保数值稳定性。因此,总内存需求如下:

模型权重(BF16):
梯度(BF16):
模型权重副本(FP32):
优化器状态(FP32):
模型参数规模
全精度训练(FP32)
混合精度训练(BF16 + FP32副本)
10亿参数
16 GB
20 GB
70亿参数
112 GB
140 GB
700亿参数
1120 GB
1400 GB
4050亿参数
6480 GB
8100 GB

(七)激活值内存:训练中的“内存大户”

在训练大型语言模型时,激活值内存(Activation Memory)是一个非常关键且复杂的部分。它不仅取决于模型的结构,还与输入数据的长度和批次大小密切相关。激活值内存的管理,往往决定了我们能否在有限的硬件资源上训练更大规模的模型。

激活值内存的大小并不固定,它会随着输入序列的长度和批次大小的变化而变化。经过仔细分析Transformer模型的反向传播过程,我们可以估算出激活值内存的大小。具体公式如下:

其中:L 是模型的层数;seq 是输入序列的长度;bs 是批次大小;h 是模型的隐藏层维度;n_heads 是多头注意力机制中的头数。

(八)激活值重计算:用计算换内存的“魔法”

在传统的训练过程中,我们会存储每一步的隐藏状态(即激活值),以便在反向传播时用来计算梯度。但有了激活值重计算,我们只需要在模型的关键节点存储少量激活值,丢弃其余的激活值,并在反向传播时从最近的保存点重新计算它们。这就好比在旅行中,你只在关键地点留下标记,而不是记录整个路线的每一个细节。

激活值重计算有几种不同的策略,每种策略在内存节省和计算成本之间都有不同的权衡:

1. 完整重计算(Full Recomputation)

这种策略会在Transformer模型的每一层之间存储激活值。因为它需要在反向传播时重新执行每一层的前向传播,所以计算成本最高,但节省的内存也最多。通常,这种策略会使计算成本和时间增加30-40%,效果非常明显。

2. 选择性重计算(Selective Recomputation)

选择性重计算是一种更高效的策略。研究人员发现,注意力机制的计算通常占用大量内存,但重新计算的成本较低。因此,我们可以丢弃这些激活值,而只存储前馈网络(Feedforward)的激活值。例如,在一个1750亿参数的GPT-3模型中,这种策略可以减少70%的激活值内存占用,而计算成本仅增加2.7%。

(九)梯度累积:用“微批次”突破内存限制

梯度累积的核心思想非常简单:将一个大批次拆分成多个小批次(微批次),然后依次对每个微批次执行前向传播和反向传播,计算梯度,并将这些梯度累加起来。最后,我们用累加后的梯度平均值来更新模型参数。这样,我们就可以在不增加内存占用的情况下,有效地模拟大批次训练的效果。


具体来说,我们把每个前向传播的批次大小称为“微批次大小”(Micro Batch Size, mbs),而把两次优化步骤之间的总批次大小称为“全局批次大小”(Global Batch Size, gbs)。如果我们每执行8次前向/反向传播后进行一次优化步骤,那么全局批次大小就是微批次大小的8倍。

虽然梯度累积解决了内存问题,但它也有一个明显的缺点:每次优化步骤需要多次连续的前向/反向传播,这会增加计算开销,从而减慢训练速度。不过,如果你仔细思考,你会发现这些前向/反向传播是可以并行化的——每个微批次的计算是独立的,唯一的区别是输入样本不同。这意味着,我们可以通过多GPU并行计算来加速这一过程。

unsetunset数据并行:用多GPU加速模型训练unsetunset

在数据并行中,每个GPU处理一个独立的微批次数据,因此每个GPU上计算出的梯度是不同的。为了保持所有模型实例的一致性,我们需要在反向传播过程中,通过一个称为“全归约”(all-reduce)的操作来平均这些梯度。全归约是数据并行中的第一个“分布式通信原语”,它负责在GPU实例和节点之间同步和通信。

(一)优化数据并行的三种策略

1. 优化一:计算与通信重叠

在简单的数据并行实现中,我们通常需要等待反向传播完成后,才开始同步梯度。但其实,我们可以将通信与计算重叠起来,让它们同时进行。具体来说,当反向传播到达最后一层时,我们可以立即开始同步这些层的梯度,而不需要等待前面层的梯度计算完成。这样,大部分全归约操作可以在反向传播过程中完成,从而提高效率。

在PyTorch中,我们可以通过为每个参数添加一个全归约钩子函数来实现这一点。当某个参数的梯度计算完成后,立即触发全归约操作,而其他参数的梯度计算仍在继续。这样可以显著减少等待梯度同步的时间。

2. 优化二:梯度分桶(Bucketing)

GPU在处理大型张量时通常比处理多个小型张量更高效,通信操作也是如此。因此,我们可以将梯度分组到“桶”中,然后对每个桶中的梯度执行一次全归约操作,而不是对每个梯度单独执行全归约。这就像打包物品时,发送几个大箱子比发送许多小箱子更高效。通过这种方式,我们可以显著减少通信开销,加快通信速度。

3. 优化三:与梯度累积结合

我们之前提到,梯度累积通过多次前向和反向传播来累积梯度,然后在最后一步更新参数。当我们将梯度累积与数据并行结合时,需要注意梯度同步的时机。在简单的实现中,每次反向传播后都会触发全归约操作,但这其实是不必要的。我们可以在最后一步统一触发全归约操作,从而减少通信开销。

(二)数据并行与梯度累积的结合

在训练大型语言模型时,全局批次大小(GBS)是一个关键参数,它直接影响模型的收敛速度和训练效率。通过引入数据并行(Data Parallelism, DP)和梯度累积(Gradient Accumulation, GA),我们可以更灵活地调整全局批次大小,同时优化训练速度和硬件利用率。

全局批次大小 (GBS) = 微批次大小 (MBS) × 梯度累积步数 (GA) × 数据并行实例数 (DP)

在实际应用中,人们通常会优先选择最大化数据并行的节点数量(DP),因为数据并行是并行化的,而梯度累积是顺序的。只有在数据并行无法满足目标全局批次大小时,才会增加梯度累积的步数。例如,当我们有足够的GPU时,可以通过增加数据并行的节点数量来加速训练,而不是单纯依赖梯度累积。

  1. 确定目标全局批次大小(GBS):通过查阅文献或实验测量模型的收敛情况,确定最佳的全局批次大小(以token为单位)。
  2. 选择序列长度:根据文献或实验选择合适的训练序列长度。通常,2-8k tokens是一个可靠的选择。
  3. 确定单个GPU的最大微批次大小(MBS):逐步增加微批次大小,直到单个GPU的内存不足。
  4. 确定可用的GPU数量:根据目标数据并行实例数(DP),计算所需的梯度累积步数。全局批次大小除以数据并行实例数,即为剩余的梯度累积步数。

(三)ZeRO:零冗余优化器,让内存管理更高效

DeepSpeed的ZeRO(Zero Redundancy Optimizer)通过将这些组件分散到不同的数据并行节点上,显著减少了内存冗余,同时仍然允许使用完整的参数集进行计算。接下来,我们将深入了解ZeRO的三个阶段:ZeRO-1、ZeRO-2和ZeRO-3。


ZeRO-1:优化器状态分区

在传统的数据并行中,所有节点在反向传播后收集相同的梯度,并同时执行相同的优化器步骤。这不仅效率低下,还浪费了大量内存。ZeRO-1通过将优化器状态分成N_d(数据并行度)等份来解决这个问题。每个模型副本只保留1/N_d的优化器状态,并在优化步骤中只更新1/N_d的浮点参数。

ZeRO-2:添加梯度分区

在ZeRO-1的基础上,ZeRO-2进一步将梯度也进行分区。由于每个副本只需要与优化器状态对应的梯度片段,因此在反向传播时,我们只需要执行“归约分散”(reduce-scatter)操作,而不是“全归约”(all-reduce)。这样,每个副本只需要保留1/N_d的梯度,从而节省更多内存。

ZeRO-3:添加参数分区

ZeRO-3(也称为FSDP,即“完全分片数据并行”)是ZeRO的最终阶段,它将模型参数也进行了分区。这意味着每个副本在需要时才会动态收集所需的参数片段,并在使用后立即释放它们。这种“按需收集”的方式进一步减少了内存占用。

unsetunset张量并行:打破内存限制的“魔法”unsetunset

在训练大型语言模型时,激活值内存往往会成为瓶颈,尤其是在单个GPU无法容纳模型的单层时。这时,张量并行(Tensor Parallelism, TP)就派上了用场。张量并行不仅分区了模型的参数、梯度和优化器状态,还分区了激活值,且无需在计算前收集所有片段。

aaamu6eru6pcXeVydZWryi6HssEZgw3GYHLOQSByFogcRM4ooEAQAiGCiQJJ5AwCJRRQjltwu7QqgWWEbSzsc57zwNW955z7u9/fvfronHvCreue3k/TnB9EZSeRKpCjl79++ONR401+GOGdLXzvp6n7DentMuidutujePdH8IZSVkJUo4dzUgtqEZU3+of//JzUxDk+wrkSUdlgZ3JdWQ+FIhKJjKg7fOEqlWp7e7uvry87O7uqqqqzszMjIyM5OXloaGh9fd0MjYezH3/P/vBdlYIjFb+oO9vxizIYDFKpdGtrC2rrlcvlxglioelolpeXx8bGWCyWWCzmHISTjWW3yqSXJrZRhCuVyp6enuzs7MnJyebmZhgMlpub6+HhYew3odPptra2ioqKEAjE2tpaZ2dnTk4O1Efmpdf8thJoNBomk1lYWDg2Nnac3psA4ZCnDiO8Dydtq1quKZqMCSm/c9OzqmDc6X5wdhIBV7vm6hiZn949QFRDX3DvD8LJaJGnC3y/Ft4uQ6QQnzpFRwYUPXYIDfPLh4Wgblx1HiSpa4voHi7x1YUTrxHhSqVSKBQKThr4fP5Jswp2d3eNb1jf1kP9us5rhvD932QtPA+XpGfOCaG+BS4PonKSiIGeCE+X5KdOca6OMELD9osQfvAbrs/uqvvtH7zrS5mEBrbT3bBP/vxtqE8hKm8Ukdzx1CkuJhj17HHCvZt+vdi9YyJcLpdjqWS74gQ4qUFo8TW2SqWanp5GoVD5+fmlpaWZmZkZGRlNTU2vpf+ameD7K5Vx1rELY3LNq65JodPpJicn0Wj0j9MQaTQaGo3m4+OztbVlMBi0Ou3Kykp9fX1uxedHrgAAIABJREFUbm5SUlJXV9fg4GBzc7OxSmlm1Wl+tF2EUygUBAJBo9EKCgrgcHhERIS9vT2FQpFIxAqFYmdnp7y8PDExcXJyUiaTdXd3Z2VlraysWPhheJqyvuhcCoWCRqOhUKjZ2ZmX/oKzgPDErd24JXboyHTI0CRsbj1xa9dsqRLLy5wkbgoi6XOB1OeRU/MJG3yzvLa2zMlhhA91aImN21mJ7U73gpwdwqoKxu/Ze9cW08looacLPDGqjkZQvlcIHyCqmyvmnO4FuzhGYmpWEqPq/T0yShED4f4Fbo9iyxFDF7+8UV04kRBV6+WagK5efl0IV6lUZDI5KSkp/qQhNjb2pFnjExISrF0V40UP5lvffxjhNIKyoWwmMaoxLqy2NGewq1XQUMpKiWmJC6upLqSbvi0yq4X3YCQxwai7N33v3vDNTe5srVx0uB1044q7h3NSQkR9bfF0amybh0uy60NYOWIYmivppT3SZTIZFovNLCkIQZe3zI5odToLivF4vIaGhtzc3IiICGhWHxwONzMz8yZ+bym0+8uc+HVV8+TWTa5+2H61Wg1NFcfn8xcXF7Oysr744oulpSWoD6ZAIJidnaXT6WlpaQUFBXQ6PTY2dmlp6a0Tx0YRDjWkZ2dnj46Ozs7Ojo2NoVAod3f3sbExAoHQ19fX1NR0+fJlZ2fntLS0vr6+1tbW/Pz817VQ92HvvsY9e3t70DJ8m5ublou1gHDYwqZLTdPNhJSb8cmOJagoxmLSzyluGeE+Hb3Xo+Psk9LtkzP8egbPIsJ7cVJMzSoimXTtsmN57tCD277F2dSuVoHbo9isBOwAUfVeIZzaLkuFNTvdD3Z+GFaU2YtCjkQFlUYFlQZ55yCSSWSMKNQ3L8g7J8AzC5nW1dcufV0I/7EeHBYWce3yQx/3JN9nydZGp/tB//aL3zg/DLU2I5T+y/NXk5NSjvlayvKz9taPHkY49A67u03Y2cKHhpDRCCoyWtjdutv/0y9U6CY3Q/gAUYWr22xBLbSgFjqaub04KbpquQW10Fq5hK/f6sVJu9t20VUrmOpVarsMOotlhAuFwurq6qqqKtb87Axnc1Oya4Fb0GCzkZGR8vLyrKysuLg4LBbL4/HUarWFXCfWX6FRVzNo7sQK3gtWKjt+yRKJBIlEtrS0bG5u1tbW/rjS2s2bNyGEGwwG6EX44uJicnJyS0sLn8+PiooaHh5+aU3s+AacLKWNIhya2AGJRPb19SkU+wPGdnZ2JiYmBAJBXV0dDodjMpk4HI5IJJLJ5ImJibLysvr6elto1nipG6DF+NBodHNzs+XRrhYQHr+yE9A37EOiPK6ovR4d79dNM1v22zLCQ55PeWBJPh29N+EpjsUV8HXeYYrbzmKjZrXwQZKmrXo5LrwyOaYxPKDwqy+u1xVPh/jmuThEwEIr7tz0rC9hDJI07xXCB4iqtuql+pLp+lIGoWGrF7eHq1tvLGO1oBZ6MKJBkqazhddQymxBzVMwYkiZ1/IunMPheHv5+T/LJjRsk5o41sVGTmFG3z//P78tQwxZl/GnE9256RsREQW9uXzpc2fjCY5EOOSpl/41Q/hP6TVDHX97Cg42TD9qTQ7td3azgPCNjY38/Hw0Gs3lctUa9a5c+tKOYzqdTigU5ubmuri45OTkLCwsvIn6N+RQnV63KuKNsJdUWs0rupjL5WZmZmIwGCKRGBsb29DQYGdn19XVBb301Ov1Ozs7xcXFOTk56+vrMpksLi6us7PzOK9EX9Ewy9ltFOEGg0EkEnV2dpLJZKlUCjVl6HQ6rVYrEAhEIhE00hTq1MZmsxsbG6empt7cjWJZRGuP6vX6zc3N+vp6KpVqoQ5hAeGJW7sJmwL4Gte9pf16dHxA77BVCE/c4EfS552KURc9fB6j6g+3wyexhTaL8KEObQ9aVJTZG+CZ5fcsvTi7vw8nxdWuxYSU+z1LK8zs7cPt1zKh+P68CzeuTQl9Ox9M2aEeJP2tT8BQhxZKYFTm9SHcP8K/xNjsYSz/pRsDRHV1If1f/t8PGstnX5r4yASOd8MiI6IBwl+A8L89AkdKZ7rTAsLX1tbS0tJwOBy00MOGRJA4gK1lDlpuSIe+rjc2NsbHx83mdbH2q/Kl6fUGg+7n3chfmuVFCcRiMRKJbG5uJhKJUVFRwcHBn3zySVBQkGBXoFAqxGJxXV1dcnLy3NycVquVyWSxsbFkMhkg/EV67jdcqNVq49QuxnSHW2N0Op1SqXzrDRpGC4+zodVqGQxGcXExnU5/0S8PCwhPYgsTNwVBA+PXY+Lv5xbGzK6ZVaMt18ITNvgeGOJn9x9+Ynf7MaruyNfhtozwoQ4NjaiCVkT+aX1ldT9eQT3of2v69fT+INz0qo+z/VoRXnoChA+SNDWvhnAngPCDn6pvAuFarXZubi43N5dIJEKVKIPBsCzk+HZV548fsVLZ4S89swFahxO8lj0qrQa/OJlAQwsVslcsUKVSVVRU1NTUCIVCuVwuEAj8/Pzm5+enpqZ8fX2JRKKbm9vFixdv3ryZlJQ0NTUVGhpKp9PfOndstxb+iv6w/exarbavr6+qqmptbe3IURYWEJ6wKQgZpt+AJd6AJQQPjsPXuFa9C49iLEYxFiOnF+/nFt1MSI1mLpn9ArDxWvhxEAWlAQh/kVYA4Tb1FfEGGtJPXguHupQjkUizEbDbUmHeWCdmfvyltfBT01auUVVM7a9UtiM9epKZ41ui0+mgaTShPlV6vV4ikUDTwVZWVu7u7kokkt2DsLe3R6PRUlNTd3Z2jl/+G0oJEP6GhD1WsdCy4jU1NT92jjicwQLC41e2HxaWnbO/eyUs6mFRRfDAuFlN2nIt3L2l/VZyxt3MvGuRsMeouvjlbYDww/rb+B7T2dlexGkL+wHCT9O/KpVKIpEolIrDjYiQGRDC02LRFIykF7dnVezBiDPh+K/O30qOaaERlIOk/Rcox4/9BGV2It7N1QeaI12hUIyNjZWVlQ0NDZm95tPqdCKlbE/1kuXITlNY5UEtPGkA++q1cIPBwOFwhoaGuFyuqZu0Wu3e3p7pHoPBQKfTx8fHjz9P15vTBCD8zWl7rJKFQmFVVVVDQ4NMZt4QZAHh8DWON7HnPqLwQW6RU0mltQiPoM89Kq95kFf8tBG935udLTwcbbsh/bg1jDNdCzedKOrwzQQQflYa0vV6/dbWFhaL7evrW1pa4nK5h1tfe3p6Pvzwwwf2QdFBqNhg62J0YIXjndA//v6C/7OcmgJ6bdGUVbG6YDLcr9D5sTuEcA6HQyQS6XT6YT5J1cqx7ZVZPtuMZ4dvzlPbs9/LTCqeE7BfyzInULfz41yd7iCc2mVaOBFAuAVxTunQzs5Obm4uDocze2YsIHz/XfjWbuKm4G/x5yPKkthCy7Vw6FV6wgbfrBOcKchfHeH7vRk0+70ZDn9hWaWsWY90CzXLw4fOLsJ1Ot3i4uLy8vKLumsxmcxvvr6WEY/D1qy1121aG71d05wcH21sbFjlC7PEBz3S/SP8f/YuvA8nrStm1BTS2yqX+nBSKl6OqV6tKaK3Vi6a9jS0/C6c0MBuq1oeJGl6MOKm8rna4qn2uk3jQAPI0a8X4fqDAPWZ1Wg0UEccaBUWuVwuk8mkB2HvGEEkEu3u7vJ4PA6Hw2azoYWxa2pqEhIS3N3dw8LCent7l5aWTHtC9fb1/ulPf/r1Lz/85E/fWhs//vCb3/7qz3/33/7+1//2x0//fMnaeO7Pl/7z3z+xu3kLQrhMJhMKhUd20NkQC+Jp6GrGwGvhpdm9dLKPOr2eKxUv7u5odNqTlXDWcwGEv30PQh3Us7Kyurq6TJ8cywg3xe3h7Zci/HAWsz2viHCZTEan07u7u3t6erq6ul5lDoT3E+EajYZIJJaXl3d1dfX19W1vb5vdqUwm889/OvfFp9evXXpy/XtXa+Mffnf+wQOHFyFcr9fLZLLt7W1oRK/ZqY0fj0R4Zf7Y4weRrg9jnjrCSrJptUVTzx4nOjtEPbgVVJE3QiP8bci+BYRTsHuP7kXcu+nXh5NlJxKd7oU73g3zdk0jNu6Y/kpzuhvm4+M3ODg4MjIyPDw8MDBApVJ7e3uhW45EIhGJxPb2diwWi0aj2ywGLBbb3t6O/ykQCAQikUg6CJ2dnV3dXd3k7u7u7s7Ozo6ODmj/4b8dHR2d+6k7ofO2tbU1NTXV1dVVV1dXVFQUFBQkJyc7ODj8+te//vzzz3Nzc03XCIFq4e6PErITidbGrAS8h0vKr/7tv/7y8eVb1z1u/eBpVbS/7nHhsx9u377/0sVG91cq66krmqTYDsIVGnXb3FhUb7NAbqMLZBgflje0ARD+hoS1rli9Xr+yspKZmTkxMaHValUqlUaj4fF49vb2lyJiD3c4N8Pt4Y9vF+EKhYJKpSKRyLq6uvb29tLS0vz8/NnZ2eO0UB0Wrra29n/9r3/xck0L9Mq1Nvo/y7709cP4uCShUGgsGeITm81eXV1dWVlZXl5eWlpaXFxcWFiYn5+fm5ubmZlhsVhMJpPBYExPT09NTdHp9MnJyYmJibGxsdHRUYgZw8PDQ0NDg4ODAwMDNBqtv7+fSqX29fX19vZSKJSenh4yef+Lv6ury/jVTyQSCQQCHo/HHQQsFovBYCDAtLa2trS0NB+EpqamhoaGioqK9PR0FxeXb775JigoqLW11RTkTCbz44/+8u2X9+7d9L9vF2Bt/OjDb16EcLVazWKxGhsb6+rq6uvr0Wj01tbWkb47EuG1xVNV+eP1pUxXR5jv04yY4Eq3R/H1JUxnh6hAT0RnCx/C8IsQPkjSwCPr//i7Lz796DKxceeZc2KQV15xZr/j3bDsRJJpRdzpbpirq3tJSUlpaWlxcXFhYWF+fj4SiczNzc3JycnKyoKm9kxNTU1OTk6yGNLS0rKysnJzcwsKCkpLS1EoVG1tbWNjIxqNxuPxnZ2dFAql7xihv79/cHDw+fPnY2Njk5OTDAZjZmYGuqmgSaBjY2ODg4MRCAQ0Nsl4T0LvwjPh7f14JTRj2vH/UvFyRHLHX8/fgkfU92DEFKzEqkhGi9Li0E+feL8U4SKFlLQ09Zy9pNPrjZa/3Q25RlU+1efSXrIDVip7u54AZ9fpdGNjY0VFRVNTU5OTk7Ozszs7O2cR4dDPESQS2dXVVVNTg0AgSkpK6uvri4uLTdsYju/xqqqqv//7f7h44c6lrx9aG7/764Pf/8dnMdEwU4TrdLqdnZ3e3l7sobAPVAx6P5oEY/2t9ajQclSASGz6t8liaGxsbGhoqK+vr6urqz0INTU1VVVV5eXlhYWFP/zww+effx4fH7+ysmLUbf9d+Dc3irP6+9pl/XiFtTHUt+DIhnS9Xj8/P5+Tk9Pc3Dw0NESlUouKiioqKo7scXkkwg8GoKvrS5gOt4ICPRD+7lnRQeW92L2UmOYH9v5Q8zg0Tv3woLJBkqYib+Tm90+dH0R9fu5aa+XSk4cx2YnEzhZ+kFdegAfCOPv9UIfW6W5YWGjE+vo6l8vl8/m7u7tCoVAsFkskEqlUKpfLlUql+ngBmmFCo9FofwrQy06oO4L+IBiVt3YDWqgQi8WWl5eTyWQej3f41ZJN9Uh/0QXq9DqVVqPW2lCTtVqvfc5ermEO2lQnuxcJ+Cb2g1r4m1D1hGWKRCISiVRbW9vS0tLY2Li2tnYWEa7T6Xp7e8vLy2tra3Nzc6uqqhwcHIaHh5FI5OEue8dRqr6+/hf/8m8o5GhT+ay1sa6E4fkkOSU50xTher1eoVBwOJzNzc2tg8A+CNsHYcckcA4C9yDwDgLfGAT7zICC8CCIfgrin4JEIpZIxMaXp9DLVKlUKpPJ5HK5XCFXKBUKpUJ5VFAoFPPz81Cdsqmpqbe3d319XaVSGRV7Q93ZFAoFGo0uLS3FYDDZ2dnx8fH19fXBwcHT09OHK+JHInyoQ4utXfd6kuLmFI9CjgZ55sYEVfTh9pJjWu7e9GutXIJq0kfWwvH1mw63At0exXu5pn70x4ulOYNPHGMhhAd65QZ45Jgh/KxM7SKXy7e3t/f29o4cPmowGM4EwgXyvda5Eer6rE5vaY504y16Chv7Q79Uih2p2HZMOoWrNj0FQLipGm9zW6/Xs9lsMplcVlaWnp6ORCInJibs7e2/CQ4PG2fFLmyaRZjJHtjCptnHmLm1xzVNvzz/11uIArOMsT9PbHYUKgq2sOlQUvnDQ8eFhQVrRdFqtSQSqbq6OjcX0dbWRqVSHR0daTQaEok0G6NyzJLfz3fhWq12dnZ2cnJydXVVKpUerre9IYQLBIKsrCwSiVRVVYXBYNLT0xsaGvz8/EZHR4+JcDJa5Pwg6vtvHPPTe9DVq7CQqqeOsKr88cf3I7xd00hNXAsN6RV5I3bXnl2/7PrZx1f/6R9/FfAs2/0RPOBZdn5aj8Pt4PQ4rFlD+llBODTJtoUb/kwgfFXEDSLXFdvSu3CtTsfgbuIWJuWa///XrQWd371DAOG24lO9Xr+0tFRaWpqenu7n5+fp6VlXV2dnZ/erL7684O3/dVCY5Xjx5wm+Cgj98M6D//7Pv/jPqz9Yzviio3+0u/3NtWsnQLhOpxsZGSkqKqqurkYgEAUFBd7e3uXl5QQC4UVVEMs+eD8RrtfroS4Rh8EJyfWGEC4UCpFIZHt7e3FxcU1NTXR0dE5OTmhoKIvFOmzJkbXwpvK5Lz69/vB2cJhfcXEWDZU36umS7PYIfs/Orzir37hEx5G1cApWgq5ebUYtJEQ13b7ujalZy4TjnR9EPXGI8XySjKvbNOvOdoYQbvkmPxMIXxfzI3ubSum9GpuphSs06hrmgFdnFfeVlzmx7CCbPQoQbkOuUalUa2trGAzGx8f73LlzERER33///b9+dv5zd68vfQOtihe8A/7L7vbf/dM//8flK1ZlNCb+/Q9231y9egKEGwyGzc3N/Pz8hoYGCoUyNTWFx+MLCwslkhOuBvh+Ivyl9+UbQjjUkF5UVNTW1lZcXIxEIrOzs9vb249cQ+hIhHe37hZn9ZchhiuRY9iaNQpG0la5XJozWF/K6sXtGRl8JMKNc7l3tggaSmcGSRoyWlRfwijPHcbUrNJ+WoAOKuT1Dip7qeBvNMGZQPiWZDea2nowwaqtvA6XqpSFEz2u+NJXX6nsjfr3zRUOEP7mtD1JyTqdTiaTMRiMtLS0Z8+e/fmjj74OCAl+To9mrVgVIxmLjqi6f/v8gl1mnlUZjYnvF5Zdd3h4MoSr1eqZmZnS0tLIyMjg4OC4uLiBgYGTyHGQByD8SOnMEE4jKKl4+cHkXJpBkoZGUPb/9BHioulHC4uN6vV6JpOZkpLS0tIyPz8/MjKSlZU1Pj5+ZAvKkQiHVlgZIEIThO0vlvXTmivGhbP2Z+Z5EcKNzezGNvNBkvqgtJ9lh7qzvZO18EGShtCwDQutLkMM0QhKGkGZn0ZxfhAZ6JXbjFqgtstzkzsd74S63I9G1+wPnTebI52Kl7egFpFp5PpSZl+7bJCkJjVz8lLJlQUTvTjpAFFNaGAXZfbXFNIPjmosLHNiduNxpeK4fnTCAE6mtpVW6z2VomyyN6K3SaZWmln7nnwECLdFR+t0OolEwmAwrly5chYHlUFLFYnF4o3NjdW11Z2dHdN+WNYqDhB+pGKmCO9s4WXCsX7P0tLj20jN3I5mbl5qR5gfMgOO7WzhDZI0tcVTQZ7ZSVH1+Pq/TZBiYYJVlUo1Pz9fWVkZFRUFh8M7Ojpe1IJyJMKN9WzLG5YRbjkvdPRdrYUTGtiujrDf/vpP0cGovnZZbdHU9ctPPJ8kOz+I8nPPyojHXfraARZW7fUk5fLFh71YiRnCEUkd175zvnzR8fNzV9PiMLja9SvfPfrqvP2lrx9EBVVUHnRK+Pav9899dAkeUU8j7I9hs7DYqOm9t6dS1DIGkWPdL11v1DTXG93W6XVChYwrl+gNtjLO7Y1e7+HCAcIPa2Ire8701C6vUcQXIfyglVVIwe5Blc6OFi5+f6lsqelayGd3draXCmiK8BRYi4dzfGJ03e0f3L3ckpCpXX7P0qOCSq5/75Sb0kFGi87/5XKoX56rU3SwN4KMFlmohRvPq9frLc/wCs0p7e1lPjvbcej70lr4cQp5VxFObZdjataePIyFhdX04qSZCYSHt0OIjTsFGZSnj+Lv2wd6uiR3tvJ7sdLPPrnaXDFvhvButLCrVUDBSNyc4v2fZSdGNX75mV13mzAnqePxgwgft4ynjnF1JczMBPynH39PbhMeH+EGg0Gl1cjVKo1OawvI1Oh1O1LRulhgO8uuGB+fU9sACD81qa0+EUA4JNmLEN7dKogKKslL66ZgJTVFk4Ge2X7P0tPj0WS00AiA9wTh/XhFVwu/rmTa71man3taP15BIyh7MGJPF3huCik3pfPyN/doBFUZYtDtcWx9yfRxEH6c+xXUwo+j0nHSmL0Lp+LlXk9SYGE1PRhxUkyL090wMlpUnvv8mXOi3VX3CP+SHoykH6/4+vytitwRM4QPdWgHiKrGslmHW0EJEQ2hPgUP7ANpRBUKOepwO+jODR+vJ6m42vXmivl//81HxMYdqxCu0emWhVzq+uy6mP925zTV6nSzfHZUb3PqIF6meU9b0Q0GA0D4cZ6vt5MGIBzS/UiE9+L28lI7vvriekRAYXvdhtfTpEDP7LzUTse7gZX5Y8bXqO8JwgdI6rqSaX+PjKvfOsSGlA8Q1b24vaKsvmfO8bXF9NiQcvvrroMkTX0p09M1oapgDCD87TzSLz6rKcIHSRoKRvzscUJ0cAUZLcxKJD64FdhWtZyTRHJ7FO90N9TDJbm9fhNfv3X+0+sNpSwzhA8Q1S0VCz5P07yepLRWLsFCqu7c8KG2y8tyBh1uBTncDvZ6koKuXqktmvrdv/+F0LhtFcLVOu0Iezmityl3tHNdLFBqNbsK6a5CKlTIxEq5WqvR6fVStVKo2F/QTKPTaXRaiUohVMiUWrXeoNfotFBK5X5KnewgpUSl0Op0Wp0OSilXq/R6veZgVTSRUg6llGtUIuX+KfabzWVinkwiVspaZ0fCKQ0jW0tHdtR4sdjv1BGAcNt1J0A45JvDCKcRlLVF9BDf3KvfOcQElzVXzN63967IG+5qFbg7w9Lj2gZ+6rr8niC8s4XbjJorzur3dU/1dE3obtstQwz4PUtLjmnqahUg07q//+5BD0ZcktP/zCW+oZTxGhHu5enr7ZqGrlrG1qxZFTHVK4jkzn/6X78qyKBYyIh5cbH21zwjwqNetAyM7T7YR1lminAKVlKGGLp5xd3ZIQqFHKsqmHC4FRjsjfRwSQr0zE2FtT24FRQXVhvshbS76t7dumuGcGLjjvsj+HdfPYgNqWwsny1I77301YP8NEq4f7H7o/joYNRTp7iU2NZAz5wfLj3pwYitQrjBYJCqlH3rM/WswRURd1nELZ7oyR4h5Y52ltP75gXbeyoFbmEib6yrdW6ULRWuifn1rCHkWNcIe0mt1a4Keflj3RX0vmnOxq5c2rXMKBjvrmUO7khFHJm4cWY4b6yrZ5UlUyvXxfzCcXIZvW+Ss7arkFLXZ4smeiqmqIXj5KQBbCm9d3tPNMtnM3kbCo36KEXfl30A4bbr6ZciPHFTEM1cipicg69xzaZJtzxHetzSlheh2wvf7dPRGz4xY7bWOFTUKy5z8hplNUP4IEmDr98K80O6O8Psr7t6PoGXZFPv3vSoLpwgtwk9XOKTouuNo4/eE4SX5w4mRdenx6N93FODvHMaylhPH0Xf+sEtMrC4AjlCaGDbX3eDR9b4e2SE+xd0NHNeF8IFAoGvr/+nH31376bvfTs/q+I9O7/vLz78u//2f1/97vGLMt6z84PikQl+9+/n4uLiTzZf0Gu8P19LUaYI72rlp8ainzyM8XBJykkidbUKMuF4P/esML/CyvzxjmZuBrw90BPh7ZpWmjMwQFSbIby5Yv7BrUD7ax4ezkkZ8Th01UqAR47bI/izx4k5iaQW1AIstPqZc6LTvbDclC5oJvZjdmczXqlSoxYppEqNelnIzXpOCqM0hPU0wGnoiZ1VoVJaRu8NozQUjpNXRNxZATt7hBROaexcnlZq1VOc9QhKUyINM7AxvyMV1bOGInub058T10T8DYkAMdoRTmlsnnkuVspn+FuRvU3x/ei+9dkdqQg7Px5LbU2gYeL622KprQ0zQ2KlXPdqE98aL+dMbwCE2677XoLwrd2o6YV72fkP8opjZlatQnjQwPjnDx85FJQ9Kq8J6BtOWDf/BZDEFtoywtHVKxH+hb5uqd9/++D2Dfe81M4njhFJ0Q3NFXOP7ocUZ1MHSWrodfh7gvC2qqW0uNa48KoUWEtj2QymZjUhsjbcvyAuvKo4m9rXLivK6o0Lq0yKamgsY0FNFBZ6pB//kVCpVEQiMS4ODo9POkGMgyWEh4fHwRJOkBcenxQbC+vv7z/ZxPvHv8bTSWmKcBpBSWri4GrXcXUbBwMK1BSspL1uk9i4Q8XL95vZsRJc3QaubrOfoBjq0JohvAcjbiybqSuari9h4GrXqe0yQsN2bfF0Y/kcuU3YT1ASG3fqS1kNpSwKdn+YvrW1cFNBlFrN9p5oRchdFnLWRLyDxnPt9p5wWchhS4QKjUqmUW1KBMtCzq5cqtPr9lSKZSF3TcQTKWUqrYYjFa8IuRsSgUKjVmrVUEquTKzR7bexLwu5qyKe8CAlXyZZFfHWRPx1MX9Lsgvx29SS93YbINx2XW8Z4fHL208b0J/dd7gSHhU5vWgVwn27qB9evho+PhM5NQ9b3Eo8tNy4LSN8qEPbi5Pu98dBzXu6JoT5IXG164gU0v1bPvfsfXzdUwmN7PetOxuNoOrBiPf7IWPF0EhiCkbc3SbsbhP24vYGSep+gry7TdiDFhnX+nwtCDcYDAqFApoi/q38VSrfkX5MpghnAwdKAAAgAElEQVQ33r3H3DBDONTb/2Cxmf0ZAqBCBklq4/ZQh8b046sgHPr21Bv2R3SZDuoy27b80fgVfLgQCxmNud7zDYBw270BLCA8cWs3eGjiJjz5ooePXWKatQj36ej71w9+99kDx2+8/b0I5MRNgdkvABtH+E/fSpr2+s2OZs4AUUXBShrLmJUFY/iGTeOL8KEO7XtSCz/md71psteFcNt9fs6UZa8X4aaOfun2qyP8TCn9rhkLEG67HrWAcNjCxn1EwfXoeLvEtBtxSaGjDLOatOV34aFDkxc9fbwI3XcyEDfikiImZs8iwk1rG8Zt00HhAOEWvr4Bwm3qyQcItyl3nCFjAMJt11kWEB4xMXMlNPLTW3f/dPnqf33z3eOKWvgqxxTDlhEeM7Pq3zMQMTnjXFlvB08JfT5lmhfattl34RawdPgQqIUf1gTaAxBuU08+QLhNueMMGQMQbrvOsoDwhA1+zMxq2CjjcUXt3cy88HGWVbXwwP6R70PC7RPTr0XCHpVVwxY2AcJt9z54gWWms7O9iNMW9gOEv0DXt7MbQnhmfDs00T3UoeGYf6ntspykjq++uJ0S22q6nroF75seGiCqcpIIbq6+TCbz7Vw8OOsrKAAQ/griveGsFhAOETdxUxA5tRA6Mm1WBU9iCy3XwuFrHD8y7UltszeBHDO7dpjfZ+JduOnX0Iu2QS38RcoAhL/hx9e64nt6ej788ENPl5S81G5kGtmqmJvS5eOW/smfvokIKN0fRl+7YVXE1KzGh9U8cfYECLfOZ7aRGiDcNvxwlBUvRfiR6IV2Wka4hYzGQ6Ah/Sif2NA+BoNx8aurKbGtLaiFtqpFa6OHc5KT46ONjQ0buqT32JShoaFz5879w//4l1/+4ve//Nc/WBd/8ft//J//+n/8b//XB7/95MvP7L783N66+JndH3//hZ3dbYDws3gDAoTbrtcAwiHf7E/t8q+/aq/fpGAl1sauVkGYb2Faao5QKLRdT5/Ish8XYv/wjx+d+/N333314NJXDtbGD37zyYMHDqeAcJlMxuPxtFqtRqPZ2tqSSqV6velYIYNOpxMI+LOzsysrK3K5nM/nSySS923KTCqV+uc///mD33z82SdXrY1/+fjKB7/95P/87//wH7/95Iu/3LhgZfziLzf+63fnAcJP9BS+/UwA4W/fBy+yACAcUgaLxf7zP/3CxSHmmXOStdHtEfzrC7fh8UkikehFOp/R/QwG49NzX9y+4e3lmuL9NNXa+NV5e4cHD980wlUq1eDgYHNzM4/HI5FIAQEBo6OjWq3WVPPNzc3S0tKoqKi4uLienh4sFtvW1iZXyE3TvPPb0LvwxKjmzhZ+V+uuVbGjmZcGw1z47AYsrLrjRNmTY5pdn3iDWvhZvM0Awm3XawDhkG9WVlZgMJifn7+/X4C10c/PPygoiEgkvhvTcJrerGeiOxuHw0EgEJWVlevr68XFxV9++WV7e7vZfGrr6+tYLLarqysxMTEtLY1IJD58+HBzc9Ossm567e/eNuiR/u759HSuCCD8dHQ+yVkAwiHVdDqdUCTkC/gni7vCXaVS+e7xwPYRrtfr5+fnw8PDe3p6ZDIZl8t1c3tKJBLNEK7T6bRaLY/HKygoyM/P39jYuHr1Kp1Of/dcZuFbACDcgjjgkAUFAMItiPOWD0EIv+Dp60umBT+nWxWDBifuFZT966ef30xHRE4vnCDeyy+57vBwYWHhLasATv8CBc4Ewqenp4OCgp4/f67T6TQajbe392GEGwyGvb09IpEYHx9PpVKVSqWdnV1XV5dZe/sLZHhHdgOEvyOOPPXLAAg/dcmPfUIej2dnZ/ePv//D73+w++Otu1bGO7+88Nf//X/+w2++/vazp55/i24/bUB7zD4+9fzM7af41PODy9cuXrkCEH5sd512wjOBcAaDERQUNDw8rNFo9vb23N3d0Wi0XC4XCARisRiqZyuVyh5KT2RkZHV19fb2tk6ns7Oz6+joAAh/0YBAs/2H50g3S2D5I5hg9bQf3dd6PoDw1yrnay0MQvivLvz1S9/Ai6GR1sZPHj35H7/5jz/csP82POYE8U93Hnx77TpA+Gt16esszPYRbjAYFhYWIiMjenp6lpaWEAjEuXPnnJyciERiXV1dU1MTtNT3+vp6QEDA+fPnnz59WllZuba2duXKlfHxcdCQbhm9xqMA4a/zuTprZQGE267HoIb0i0FhoWPM2PkNa6M7rvN3127cyimwNiOU/kEJ6vpDR4Bwm70/zgTC+Xx+UVFRWVkZh8NhMBj9/f3Dw8Pr6+v5+fmNjY3QOmMymYzBYNBotOHh4fn5eTKZfPv27fe8O1svbi8/nVJfwjSu2UNGi8oQw22VyzSCsqGU5e+eFRVYTkaLDi82OkjSdLcJWyuXOlv40IpkfThpa+VSe/0WjaA8WOhvD1OzSmriQFO5gVq4zT7jxzEMIPw4Kr2dNK/SnS2JLfQh0/5gd/t+UYVxtharNmxnape3o77Nn9UM4YMkDb5hK9gHUVc8RW2XV+Q9d3scc/+WT0kujUZQ5qV03vrBzd05vr6UAS06eTqzs2m12omJicbGRmg4uP4gGAwGgUAgkUhM69nGQzgcDo/Hv3sjCCzfUKbvwjuauZ5Pkn/zyz9GB6Oo7XJotZ6MeNylrx2Topor88cvX3R0fxR/43s3++ue+xOyElW5KV1ff3EnJbZtgKQuzKB++9X9jz68eP7ctbzU7raqlcsXH35+7urXX9yK8C+pK552fxT/6Z+/+9MfvsxN6RwgqgDCLbvGxo8ChNuugwDCbdc3NmCZKcIHSRoKVhzmX/DJR3/NTeloKGW6OcfGhqGykvCpsJb2+q2/fPJNflp3iG+un0dGZwtvqEN7Ogg3GAx6vd6qeVp0B8GU7jYg9hs3wRThPRhRZf74w1tBsLBqartskKRprVqyv/rsDx98nhDREOiZ6+IQS2rmdreJ/vLRpbaqZTOE42o32qqWO1t4Lg4x/h7ZyTEtX1+43dnCy0kkPnGICfDMcXGIRiFH4sJqv/7iDgUjBgh/4959kycACH+T6r5a2QDhZvrp9Xpoki+1Wq04CGY9njQajUqlgpih0WjebQyYIrwfryjM6Ll88d6tH9wzE7CZCTjnh+GJUfWFmb34BnY6HHP10sMBkhqVP+L+GFZbRD9NhJs5EXw8UgFThA91aKl4udeTFFhYTV+7rLtNGOCJuHvT74fvnyZGNj5zTowMKOvBiPvxiq8+t0chR80QfrDwrhpTvep4JwQWWl2URf3Lx5dhIdVuj+BO98K9XFM8n6Rga9Yby2b/4zcfkZp2AMKP9MhZ2QkQbrueAgg3841CoWAymbOzs3Q6HY/H43C4paUlI8UlEsnQ0FBfX59UKuXz+ZOTk3t7e2YlvEsfjQinEVUNpYz79t5RgSX37L3jI6qjgkpuXHF+5hL/8I5/XERVRGCR/XXXQZKmvpTh6ZpQVTAOEG5rd4IZwvtwUg+XpNjQyh6suDz3+aWvHe7ZBXx94Y6nS/KTh7F+bpkdzdzOFv4Xn16vLZoyQ/ggSYOr3QjxyX/yMKahbKY4m/bXz+2cH0TeueHz6F6455NkL9dUbM1aQynrg99+QmhgA4Tb2s1glT0A4VbJdaqJX4rwxK3dxE2B2TKjxhfelt6FQxk3BRay29q7cL1ev7m5mZ6eTiaTCQRCdXU1HA6vqKiQSCQGw/4823Q6PTw8PCIigsPhrK2tFRQU0On0U3XY6Z7MFOEJkbU3rjr7uKX+9cIPz5zjwvyQjx+EFGX2xYSUezyBZycRLl28Q2zcQaZ1PXOJbyyfAQg/XV+9/GymCO/FSasLJm9d93JziqstmqotmvZ1z/RwSbrw2c2nTnExwagH9gHpcdj48Lrvv3EiNXHNEN7Vwg/2Ql759lFcWHVb5VJcWM31y08q88cTIuof3gn2dk176hiHSO6MCiq/eOEuuU0IEP5y99hwCoBw23WOZYTHLbMDqSPP0ATfzr6YmZXELYER3tCGBYSHPqe7t7a7t+CftRGD+kfjV3bM8trUYqOQh/R6/ejoqL+/P5vN3tnZ4XA5FRUVGRkZAgEfeuG6trZWVFQUGRm5vb0tEokKCgowGMw73JZuivDa4qnclM5UWOs9e6/Y0IqSHFqgV1ZEQKGXa2JCVB2xcfuevU9sKCrAIwMWWtHVKgAIt7XH3hThHc08eET93Ru+TnfD0mCYHoyY1MRpLJ+NCUYVZPTiatdhIVVQdTwTjt9fU/zn3dlaKubv2fndvOLm65aek0QqyqQ+uhsW7JPv8zQtwr+kOKs/zLfIwznJ4XZQUlRzP14JEG5rN4NV9gCEWyXXqSa2gPCEdZ43iXITnnIzPvleNjKgbzhhg2eGYQsI9yFR7mQg7BLTvvX2v5edHzu3bpbXBhGu1WoJBEJwcLBWq9XpdMvLyxkZGWg0GhqYZDAYtFptX19ffHz89va2Wq2urq6urKw0Hj1Vz53KyYwIhzoV9+MVXa2C4mwqunqZghVXFYynwlrT49HY2rUBohqFHEmKbsiIR7dVLQ2Q1ADhp+IiK05iivB+vAJXt9FUPt9csYCvZ0MjCGgEZWcL76D3mbqrVdCMmm9GLfTi9g4PKutq3S1DDBdn00oRQy2oBXKbsK54uiRnsDxvBFu70YvdH1FWihgqQwx3twoGSRqAcCv8ZHtJAcJtzyc/WWQB4bHzG3ezkHYJqV4EckDfcDRrOWHTilp43BI7anrRC999KSDkMaouYd0c/zaIcI1Gg8FgwsLCtFrtxsZGUVFRfn7+5ubmT2rt/+/v74cQrtFo6urqCgsL3+HX4aYIh2b5GCBpqHjFwUhiTT9BScFKerF70NhfGlG1/xH3t48A4aa3jS1smyLcOGeLhY3BDo3xqFktfJCkoRFV+7VzgvLA+5oBkvrgowr6NbCfYP+QauigEIBwW7gBTmwDQPiJpXvjGS0gPII+961/0EfXb15wdv0BlhgyTE+0BuFJbCF8lfMYVfd9UFjExMzhKrgNIlyr1VIolMDAQC6XW1BQ4OXlVV9fz2AwBAIBBoMRCoU7Ozvl5eWenp40Gk0mk1VVVaFQKKVK+cb99JZOcBjhxu/042yc2qCytyTPGTuttQg3dbEZwk0PHWcbIPyM3Ss/Nxcg/Od62NInSwifnPnOP+haJMy9BXctCuZS0xS/yjEjsYWG9MSt3fBx5kVP38eo2sPsh8qxwe5sU1NT/v7+LBartLQ0ODgYBoMhkciFhYWAgICFhQUajRYVFRUcHFxUVLS2tpaXl4dGo9+Hd+HGCbyO831tTAMQbkvPugEg3KbccYaMAQi3XWdZQHjs3Jp9csZNeMqzNsL16HiXmia4NQiHr3Ed8ku/8faLmV01A7/xow0inMvlFhUVPX/+XCaTiQ/C3p5ELpfX19eLRCKlUvnTzr2lpaXCwkImk2m73n1ly0At/JUltKECAMJtyBlnyhSAcNt1lwWEJ2zwvQjka1Gwy0Fht5IzggfGD1emLdTCYYtbV8Oin7URjMA+vGFrCIc6rG0fBDOfHa5qi0Si5eVltUZtlvJd+ggQ/i55EyD8XfLmaV4LQPhpqm3duSwgfP9l9ho3fJwVSH0exVhM2OAfZrAFhCds8EOGJuFr3MO5jHtsEOHWyfeup2Yymd99c6Mws68Xt0dtl1sbQ7zznRwfb2xsvOs6nY3rgxCeHN3S1Sogtwmtip0t/PQ47IXPbsaF1Z4ge1erICW2xfWJ97vdanU27gPrrQQIt16z08phGeFG1r5owwLCX5TFdD9A+Gn5+YTnYbFYF774xu1RfFxYTXx4rbXxxuWnTk6PzLr0n9AUkO2VFaDRaOfPn//2qzuOdwOd7gVbFR3vBl66eO/Xv/zD11/aO94Nsiqv071gx7tB339z//GjJ7Ozs698HaCA01YAIPy0FT/++QDCj6/Ve5iSzWY/fep2+dK1K9/fOEH85uKl1NTU3d3d91A6G7xkHo9XVlYWFhYRER4VEWFlDI8KD48ICws/cfaoqOjGxkahUGiDygCTLCsAEG5Zn7d5FCD8bapv8+fW6XS7u7s7OztQ/4AT/JVIJFatIWbzkpxhA/V6PbRIj+othXd+WaAzfHNYNB0g3KI8b/UgQPhblR+cHCgAFAAK2LoCAOG26yGAcNv1DbAMKAAUAArYgAIA4TbghBeYACH8Cw9v747ewIFxa6NjZcNvv710FZ5ibUYovV16zrX7DxYWFl5gHdgNFAAKAAWAAm9ZAYDwt+wAC6cXi8VPnz797V8++/j2/U8fOFob/+vq9X/49W/+/cJXnzo47sfjlAClPEj8h28v3XN0XF1dtWAhOAQUAAqcpgJ6vX5paWlnZwfqxCAUCul0ulwuhxbr43A4vb29JBKJwWCIxeLl5WXQXfE0vfNWzgUQ/lZkP9ZJNRoNhULJy8/PyUPmIK2O2Xl5GVlZWQjECfLmIJG5+flEEukdXibkWD4AiYACtqSAWCzOy8ujUqkajUan05WVlfn6+m5tbRkMBrlc3tXVlZ6enpaWFhERMTo6WlpaSqFQdDqtLV0BsOU1KwAQ/poFfb3FqdVquVwue0tBpVIdnvjs9V4gKA0oABQ4vgITExPR0dGjo6NarXZqaur8+fO3b99eXl42GAwajWZ7e3ttbW1+ft7Dw4NIJGZnZ+fk5EgkkuOXD1KeOQUAws+cy4DBQAGgwPuogF6vb29vj4uLW1pekivkFy5cyMrKevz4MYRwqC1dr9ePjo46OjpOTk42NTXBYLDt7e33Uaz35poBwt8bV4MLBQoABc6yAjqdrrW1NTk5eX19PTw83MXFpbOz8/79+1NTU1rtfmu5TqdbXl52dnYuLS3d29vr7Oz08/NbXFw8yxcNbH+JAgDhLxEIHAYKAAWAAraggF6vx+FwSUlJDAbDxcXl1q1bV65c+eCDDzIzM2dmZvh8/vLyspubW3Bw8PT0tFgsxuFwgYGBoEeqLfjuzdkAEP7mtAUlAwVOQwGtViuXy6EuyjqdTi6Xq1Qq44l1Op1EIpHJZAaDQaVSKZVK0L/BKM6Z2xgYGIDBYFNTU3w+n8PhjIyMwGCwkZGRhoYGAoHQ1tb24MEDOBxeXl7+/PnzwsLCxMREPp9/5i4TGHx8BQDCj68VSAkUsEUFNjY2qFQq1GuJz+fj8XjTJadWV1fLy8uJRKJOp1tYWBgZGYHGINnilQCbXqbA9vZ2amoqjUbTaDQGg0Emky0tLXE4nI6OjsHBwdXV1cHBQRqNNjQ0NDMzg0AgGhoalErly0oFx8+wAgDhZ9h5wHSggF6vb2xsLCoqEolECoWivb3d2dkZi8VCyuj1eqhmFh0drdPpJiYm8vPzjb2fgHpnTgGNRtPf3z8/Pw+9/Ibs12g0AoFAKpWatq/I5fLBwcGlpSXTnWfueoHBL1UAIPylEoEEQAHbVUCtVicnJzc2NqpUKiaTGR4eHhIS0traarR4cXEhIiIiJCREr9fv7Ozk5OQMDw+Dr3WjPmduQ6FQHGe0p06nUygUUGX9zF0jMPj4CgCEH18rW0/J5XGhyZj0er1YLGaz2dADrNfrpVLp7Ozs5OQkn8/XaDRzc3PgS9zW3Xk8+4RCYUREBIFA4PF4MBisuqYaVYlqaWkx+len06WmpoaEhBgMhr29vby8vK6uLtM63PHOA1IBBYACtqgAQLgteuUENmm12vT09OHhYYPBIBKJqqqqEAgEtACwVCqtqqqKjY0NCwvLyMjY2tqKiIhYWVk5wVlAFltTQCAQRERE4PF4AoHw8ccfe3t737p1y9vb2/RXmhHhUqk0Ly+PQCCAypmt+RHYAxQ4mQIA4SfTzeZycTicmzdvrqysaDQaGo3m6uoaFBTE4/EMBoNarZ47CH19fUFBQTMzM1lZWRkZGTZ3DcAg6xVQKpUJCQlYLHZzc7Ozs7OjoyMhISEnJ2dwcLC3t1ewK5iYmPD29nZ2dh4bGxMKhXl5eb29vWCZcOuVBjmAAraoAEC4LXrlBDYNDw/fu3dvb29vY2MjNzcXBoOlpKRACNfr9VqtVqVSkclkGAy2urra19fn4PAAtKaeQGdby6LX6/Pz88vKyqRSqVqtVigU0wdhaGiourp6c2uzu7u7oKAAiUSSSKTZ2dmMjIzJyUlbuwpgD1AAKHAyBQDCT6abzeXCYDBubm58Pr+iosLb2xuBQISFhTGZTOidqFarnZ6eTk5OxmAwMpmMwWBcuXJFJpPa3GUAg6xXYHJysqamBhr+C/1c02q1bDabxWLJZDKNRqM+CCqVanR0tLGxEQwUtl5jkAMoYKMKAITbqGOsNau7u9vV1XV7exuJRIaGhgYEBDg4OOBwOB6PJxKJ5ufnk5OTkUjk4uKiWq2m0+lXr16Fpvuw9kQgva0poFart7e3TadzOdJCvV4vFAoFAoGxp9uRycBOoABQ4AwpABB+hpxlydS5uTl7e3uBQLCzszO/MI9rx6Wnp4+Pj1dVVWGxWBQKZW9v7+7uHhsbOz8/RyAQnjx5AhrSLQkKjgEFgAJAAZtXACDc5l10PAOlUumTJ0+mp6ehBYv2pHs7OzsisQiDwXR3d7PZbCaTyWAwWCyWWCyOjIysq6s7XsEgFVAAKAAUAArYqAIA4TbqGGvN0uv13d3dExMTphm1Wu2edA+aUFP/U1Cr1UVFRXt7e6YpwTZQACgAFAAKnDkFAMLPnMteaLBWqz3mYCG1Wv3CUsABoABQACgAFDgjCgCEnxFHATOBAkABoABQACjwcwUAwn+uB/gEFAAKAAWAAkCBM6IAQPgZcRQwEygAFAAKAAWAAj9XACD853qAT0ABoABQACgAFDgjCgCEnxFHATOBAkABoABQACjwcwUAwn+uB/gEFAAKAAWAAkCBM6IAQPgZcRQwEygAFAAKAAWAAj9XACD853qAT0ABoABQACgAFDgjCgCEnxFHva9m/jSn3N/+v28yWLskibXp3zc9wfUCBd4xBQDC3zGHvjuXo9frZTLZ1tbW+k9he3tbo9Gc/hVqNBqJRCKXy2UymVAoNJsCTy6XQztFIpFMJnsViGo0GpFIpFAodDod9Jtla2vr+KvRqFQqLpf7Kvrw+XylUqlSqXZ3dxUKxasU9VryKpXKH6f0hyRVKpW7u7svXZDttZwXFAIUOCsKAISfFU+9d3bqdLqFhYWamprU1NSoqKiCgoKmpiap9LhrnB9G6eE9Rk0tHDIYDAKBoLu7e/IgtLW1QdPTQln0ej2TxUSj0XK5vL+/f2FhASKuaYHGbeOG8bxmG2KJeGBgYH5+fnNzUygUKpXKmJiYI9eEhQBvml2v129tbRUXF5vuPLxtZoPZx/Ly8sXFRS6X293dvbq6CmU3S2Ms80gbTI+a/daxXJoxo+nGwsJCR0cHJCmbze7u7j7Ob5QXGWxaMtgGCrwbCgCEvxt+fAevQq/X7+4KGIzpvLw8JyenwcFBFovF5XJYLNba2tre3h6Xy11aWpqent7e3tZqtbu7uwwGY35+Xn4QFhcX5+bmRCKRVCrdX4B1fp7NZhsMBq1Wy+PxflybdWZmZmdnR6PRCAQCBoMxOzsrlUpFIhGbzZ6fn+dwOKurq1NTU5ubm0tLS7m5ue3t7TgcDgaDKRQKpVI5Pz8/MzMjEokoFEpycrJUKqVSqYuLiyKRaHZ2lsFg7O3tyWSy+fn5ubm51YMA2QMBRiKRcDgcnU63s7MjFAm1Wq1AIFhbW+vr6yORSEgksrm5eX1j/fz58ywWa3p62oguvV4vlUpnZmaYTKZIJIKMX1xcVCqVs7Oz7u7uGo2Gx+NJJBKlUsnj8RQKBYvFWl1dZTAYW1tbCwsLy8vLSqWSz+fPz8+zWCw2m21s23BzcxsYGODz+VQqdW5ubmNjY25ujsViSSRinU63vbNNp9M3NzfVajWkNoPBEAqFcrl8fX19dnaWy+Uaf8EIBILFxUUjTfV6/cbGxvJBkEgkKysrDAaDz+crFIqNjQ2NRiMUCnk8nkqlYrPZxpr30NBQbm4uZB6bze7t7eVwOJubm4uLiywWa3d39+Am2aXT6UtLS0qlUiKRzM/PQyUrlcrV1dWFhYXNzU2jGe/gcwIu6f1WACD8/fa/bV+9Xq/X6XQ4HM7T01MikfB4vMrKyuzs7LKysp6envLy8vT09IyMjLS0tI2NjaampvT09Orq6pmZma6uLjgcnpeX19zc3N/fn5mZmZSUNDo6ajAYZDJZa2treHh4dnZ2UVHR7OwshUIpKirKzs7u6Oggk8mxsbEIBOL58+elpaX5+flFRUUkEikzMxODwbS1tUVGRu7t7fX29qakpBQWFrYdBDgcLtmToFAoCMDZ2dmFhYXT09N0Ot3HxweBQMDh8IyMjPj4eCKRqFQq9Xr99PR0eXm5VCotLi6uqanZ2tqqrq7G4XAVFRUIBMLT0zMqKopOp3/00UcZGRk5OTnFxcUCgcBgMEilUgwGk5qaWlxc3NHRAVmekpJCJpMZDIaLi4tItL/C7MjIyOrqanNz88LCgouLCwKBSE5OjoyMTD8ITCazqakpKioqMzMTiUTu7OxAkHv8+DGVSl1bX0OhUGg0OicnJz09PTo6uqura31jPTMzs7CwsKCgYGpqikqllpSUpKWltbS0MBiMtLS01NTUqakpCLdqtXp0dLSiokKlUkF1ca1Wm5OTExkZ2dDQ0NHRkZ6ejkQis7OzmUwmAoFYWVlpaWkpLi5eXl4uLi6enJyEcg0MDGRlZUFlzs3NVVRUzMzMIBCI1NTU+Ph4NBrN4/Egq4qLiykUyvj4eHV1dWpqaktLy+LiYmhoaGZm5vPnzwHCbftBB9adXAGA8JNrB3KejgJ4PN7Ly0smk01PT9+9ezc9PT0qKiohISE4OLiyspJOp9+9e3doaCg2NjYyMpJCoUxOToaEhJSUlFCpVAi0vr6+jY2NQpHQYDBIJJLCwsg8lrYAAAqvSURBVEIYDDY4OJiRkdHa2jo0NFRdXR0ZGRkXF4dCoZ4+fUqlUtnb7Obm5vz8fF9f3/z8/IyMDCPCdzg7z549CwgISE9P9/X1LSoqgsPhIpEoISGhoaEhLy8vLCwMi8VubGx0dXU5Ojr29/dHRUWVlJTk5OTk5uaKRCKDwTAzMxMSEkImk318fJycnHp6ekJCQgYGBlJTUysqKtLT09va2nZ2dj755JOOjg4ajRYWFjY5OanX61ksFgwGw+PxKysrXV1doaGh4+PjTU1NAQEBY2NjLi4ufD4/Pz+fRCJNT0+npqaOjo5++OGHZDK5tbXVwcEBg8HExcU1NzenpqYmJCT09fXFxMTMzs5CyIQQzmKxIiMjc3Nz4XA4kUjMzMxMS0v78efOrVu3ioqKPDw8WlpaxsfH6+vrQ0JCIiMjSSQSDAbr6OgQi/cr6waDIT8/PzAw8P79+6WlpbOzs1qtVqPR+Pn5paamjo+Pp6enFxYWzszMuLm5NTY2wmAwFAoVGhrq7OxcU1MTGxvLYDAg6JoifGxsLDo6enh4OCIiorGxsaqqKikpqbGx8eLFiyUlJTAYLCMjg06nt7W1hYeHR0ZG9vf3X7t2jUKhiMXi07lRwVmAAqevAED46WsOzmidAkaEj4+P37lzp6GhobGxsampCZ4AJ5PJMpns/v374+PjVCo1Pz8/Ojq6pqbG398fh8PNzMykpqbm5OTExsZOTExAbbwSiaS4uDgnJ2dpaQmJRBYUFGQehMTExOjoaBQKlZCQwGazmUymp6dnaWlpUFBQbm5uWlpaU1NTW1tbQEDA8vKyo6Njenp6c3MzZExMTMz29jYcDsdgMKOjo5WVlYGBgU1NTV1dXYGBgVtbW0gkkkQiNTQ0/FjD3t3dNRgMu7u7GRkZDg4O6enp/v7+CQkJSUlJGxsb6enpjY2NUHqBQHDu3Dkej7eyshIXFzc0NKTX66empuBw+MjIiEwme/78OQwG29raolKpT58+HRkZcXJy2tzczM/Px2Aw/f39MTExIyMjH3744c7OztDQUEBAAIvFyszMrKyszMjIKC8vX19fT0lJYbFYEHqfPHmCx+OnpqYiIyPz8vJyc3Pn5+dra2thMFh8fLyTk1NTU1NVVdXw8HBubm5eXl5cXFxYWBgOh0MgEMvLy8bK7uDgYGlpaWho6PDwMI/H0+l0Go3G398fi8Wy2eysrKzm5maBQODl5dXQ0FBZWfno0aP4+Hg4HP70qWteXp5AIICKotFoqampYrFYJpMNDQ1BCE9ISBgbG6NQKHFxcQgE4vvvv29pbWlqasLhcLW1tVlZWdHR0aGhoRQK5f79+9vb20arrLvtQGqgwFlQACD8LHjp/baxp6cnOjoaeuEKg8Xm5+fX19e3t7fn5eUNDg7K5XJPT88fQYXH4ysqKkJDQ1taWioqKuBwOITq5pbmvLw8I6UkEgkSiXRzcysuLk5JSSEQCElJScnJyTExMcnJyVDNm8vl0ul0Z2fnwsLC0NDQuro6qNEYi8W6urpisdjS0lIYDFZbW9vU1NTZ2RkSEoJGo5OSknA4HIVCqa+v9/f3Ly8vHxgYiI+P53A4KBSKQqHgcLjy8nKoMUCj0RAIhBs3buDx+La2Nnd394GBAQ6HU1hY2NXVBSF2cnLy8uXLIpFoY2PjR1aNj48bDAYejwdV6FtbW9vb25OSklAoVGJiIvRSwNnZGYvFolCotPS09PT0kJCQycnJ7777jsfjjY+Pw+HwxcXFkpKS1tbW0tLSpqamra2t/Pz8+fl5COF5eXkhISFYLDYjI6OysrK8vHx5eRmDweTk5JDJ5MDAwKqqqoqKiuHh4fT09MzMTDgcnpiY2N3dDf0aMMJSrVZPTU01NzcbG9I1Gk1sbGx3d7dUKkWj0WlpabW1tX5+fouLi4ODg8+ePautrSUQCB4eHjgcTqlUQrf80NCQn59fXV1dW1tbfX19RkbG5OQkAoGYnp4eGhrKyckZGRnx8PBAVaLq6ura29tRKFRycnJsbCz0i8fHx4fL5Rqter8fI3D176YCAOHvpl/fpatis9kjIyMajUatVi8sLODx+J6ensXFRSaTCfVH6+vr43K5k5OTaDSaTCYLBAIul0smk4lE4o9vgtlsNovFEgqF0Fe5RCIpKSkODw8nEAjj4+MSiYTFYmFx2J6entHR0aWlJRaLpVAoJBJJd3c3rh3X09OzsrKyuLhIIBDm5uY6OztJJBKfzyeTye3t7VNTUzwej0ajUSgUGo22trY2NzfX3t7+I+x3dnZ4PN7o6KhcLp+ZmYE6c/3Yfm4cFrW9vd3f3887CBQKRSaTyeVyJpO5vb29urqKx+NnZmagDvBSqZTJZPJ4PIPBoNPpNjc38Xg8gUBYW1ubn5/H4/EdHR08Hlcmk/X29vb39y8uLvb09JDJ5OfPn+/u7qLRaIVCweVyR0dHxWLxzMzM+vr63Nzc4uKiVCqdnp4WiUSQOBwOp6mpqb+/n8ViQZ3dRCLR6uoqk8mUSCSTk5M4HI7cQ+ZwOAsLC7h2HIlEgnrJzczMSCQS07tOJpMZK9OQ2cPDwxsbGzqdbnd3t7+/H4vF0ul0aBzd6Ojo6uoqh8MZHx+HevlBRW1vb7e1tWEwGAKBMDAwsN9PTcCn0+l8Pp/D4TCZTKlUOjs7297e3tHRsbq6ura21t3dTSKRoHJoNJpCoQAIN/UL2H7HFAAIf8cc+g5eDtSpDbowvV6v0Wi0Wq3uIEDfzlqtFkoDHYIGO0HvX6EB1tBfqAS5XE6hUMjkbo1GYzwKZTQtVq/XQyUYd0LpoZ3Go1AJ2p+C0QyNRgOZYTwFZLCpJVBio7UGg8F0j/F0pvtNRYASQG3UkAJQf3vIYMgi6HRQdzBj4aYmGXealmy85MMpjQobz2tMY0ZK6LqgYqG/ULHQ5Ri9Y7y6I8uB3K05CFB2o8HQBvTjAEoAnVGz/9r9b56FXp2Y2gC2gQLvmAIA4e+YQ8HlvEQBnU63dxBekg4cBgoABYACNq8AQLjNuwgYCBQACgAFgAJAgaMUAAg/ShWwDygAFAAKAAWAAjavAEC4zbsIGAgUAAoABYACQIGjFAAIP0oVsA8oABQACgAFgAI2rwBAuM27CBgIFAAKAAWAAkCBoxQACD9KFbAPKAAUAAoABYACNq8AQLjNuwgYCBQACgAFgAJAgaMUAAg/ShWwDygAFAAKAAWAAjavAEC4zbsIGAgUAAoABYACQIGjFAAIP0oVsA8oABQACgAFgAI2rwBAuM27CBgIFAAKAAWAAkCBoxQACD9KFbAPKAAUAAoABYACNq8AQLjNuwgYCBQACgAFgAJAgaMUAAg/ShWwDygAFAAKAAWAAjavAEC4zbsIGAgUAAoABYACQIGjFAAIP0oVsA8oABQACgAFgAI2rwBAuM27CBgIFAAKAAWAAkCBoxQACD9KFbAPKAAUAAoABYACNq8AQLjNuwgYCBQACgAFgAJAgaMUAAg/ShWwDygAFAAKAAWAAjavAEC4zbsIGAgUAAoABYACQIGjFAAIP0oVsA8oABQACgAFgAI2rwBAuM27CBgIFAAKAAWAAkCBoxQACD9KFbAPKAAU+P82GgKjITAaAoM+BEar8EEfRaMOHA2B0RAYDYHREBgNAWwhAADak9RGVt+hhwAAAABJRU5ErkJggg==" alt="" />

(一)张量并行的基本原理

张量并行的核心在于矩阵乘法的数学性质。在神经网络中,矩阵乘法通常表示为 X × W,其中:

  • X 是输入或激活值;
  • W 是神经网络层的权重。

张量并行利用了矩阵乘法的两个基本性质:

  1. 按列分区:可以将矩阵 B 的每一列分别与矩阵 A 相乘,然后组合结果。
  2. 按行分区:可以将矩阵 A 的每一行分别与矩阵 B 相乘,然后将结果相加。

(二)按行分区(Row-wise Sharding)

按行分区是另一种实现方式。具体步骤如下:

  1. 按行分割权重矩阵:将权重矩阵 W 按行分割成多个片段,并将这些片段分配到不同的GPU上。
  2. 分割输入矩阵:将输入矩阵 X 分割成多个片段(需要一个“分散”操作,scatter)。
  3. 计算局部结果:每个GPU计算局部输入矩阵与局部权重矩阵的乘积。
  4. 汇总结果:通过一个“全归约”(all-reduce)操作,将所有GPU上的局部结果相加,得到最终结果。

(三)多头注意力机制(MHA)中的张量并行

多头注意力机制是Transformer模型的核心部分,它涉及多个矩阵乘法操作(Q、K、V)。张量并行在MHA中的应用也非常直观:

  1. 按列分区(Column Parallel):将Q、K、V矩阵按列分割,并分配到不同的GPU上。每个GPU计算一个或多个注意力头的输出。这种方法非常适合多头注意力机制,因为每个GPU可以独立计算一个或多个头的注意力结果。
  2. 按行分区(Row Parallel):对于输出投影(Output Projection),可以按行分割权重矩阵,从而减少每个GPU上的内存需求。

(四)序列并行的基本原理

序列并行的核心思想是:在张量并行的基础上,进一步将激活值和计算分割到不同的GPU上,但这次是沿着输入序列的维度,而不是隐藏维度。这种方法特别适用于那些需要完整隐藏维度的操作,如LayerNorm和Dropout。

以LayerNorm为例,它需要完整的隐藏维度来计算均值和方差:

其中,均值 和方差 是在隐藏维度上计算的。尽管LayerNorm的计算成本较低,但它仍然需要大量的激活值内存,因为需要完整的隐藏维度。序列井行通过在序列维度上分割激活值,将内存负担分散到多个GPU上,从而显著减少了每个GPU的内存需求。

请提供真实信息以便我们与您联系
公司信息
联系人信息
留言及疑问