shadps4-emu/ext-cryptopp
1
0
Fork 0
mirror of https://github.com/shadps4-emu/ext-cryptopp.git synced 2024-11-23 18:09:48 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3af3054e87
ext-cryptopp/adv_simd.h

1282 lines
54 KiB
C
Raw Normal View History

Rename files with dashes to underscores (GH #736) Also see https://groups.google.com/forum/#!topic/cryptopp-users/HBz-6gZZFOA on the mailing list
2018-11-10 13:00:14 +00:00
// adv_simd.h - written and placed in the public domain by Jeffrey Walton
Update documentation
2018-07-02 02:25:07 +00:00
Rename files with dashes to underscores (GH #736) Also see https://groups.google.com/forum/#!topic/cryptopp-users/HBz-6gZZFOA on the mailing list
2018-11-10 13:00:14 +00:00
/// \file adv_simd.h
Update documentation
2018-07-02 02:25:07 +00:00
/// \brief Template for AdvancedProcessBlocks and SIMD processing
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
// The SIMD based implementations for ciphers that use SSE, NEON and Power7
// have a commom pattern. Namely, they have a specialized implementation of
// AdvancedProcessBlocks which processes multiple block using hardware
// acceleration. After several implementations we noticed a lot of copy and
Rename files with dashes to underscores (GH #736) Also see https://groups.google.com/forum/#!topic/cryptopp-users/HBz-6gZZFOA on the mailing list
2018-11-10 13:00:14 +00:00
// paste occuring. adv_simd.h provides a template to avoid the copy and paste.
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
//
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
// There are 6 templates provided in this file. The number following the
// function name, 128, is the block size in bits. The name following the
// block size is the arrangement and acceleration. For example 4x1_SSE means
// Intel SSE using two encrypt (or decrypt) functions: one that operates on
// 4 SIMD words, and one that operates on 1 SIMD words.
Update comments
2018-08-14 09:15:32 +00:00
//
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
// * AdvancedProcessBlocks128_4x1_SSE
// * AdvancedProcessBlocks128_6x2_SSE
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
// * AdvancedProcessBlocks128_4x1_NEON
Add CRYPTOPP_ARM_ASIMD_AVAILABLE define for Aarch64
2020-07-29 03:06:34 +00:00
// * AdvancedProcessBlocks128_6x1_NEON
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
// * AdvancedProcessBlocks128_4x1_ALTIVEC
Update comments
2018-08-12 05:12:00 +00:00
// * AdvancedProcessBlocks128_6x1_ALTIVEC
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
//
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
// If an arrangement ends in 2, like 6x2, then the template will handle the
Update comments
2018-08-15 00:49:26 +00:00
// single block case by padding with 0's and using the two SIMD word
// function. This happens at most one time when processing multiple blocks.
// The extra processing of a zero block is trivial and worth the tradeoff.
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
//
// The MAYBE_CONST macro present on x86 is a SunCC workaround. Some versions
// of SunCC lose/drop the const-ness in the F1 and F4 functions. It eventually
// results in a failed link due to the const/non-const mismatch.
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
//
// In July 2020 the library stopped using 64-bit block version of
// AdvancedProcessBlocks. Testing showed unreliable results and failed
Update comments
2020-07-07 19:37:37 +00:00
// self tests on occasion. Also see Issue 945 and
Update comments (GH #945)
2020-07-07 19:32:48 +00:00
// https://github.com/weidai11/cryptopp/commit/dd7598e638bb.
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#ifndef CRYPTOPP_ADVANCED_SIMD_TEMPLATES
Fix ODR violation in AdvancedProcessBlocks_{ARCH} (GH #585) The ALTIVEC function required an inline declaration. Lack of inline caused the self test failure. Two NEON functions needed the same. We also cleaned up constants in unnamed namespaces
2018-02-20 18:17:05 +00:00
#define CRYPTOPP_ADVANCED_SIMD_TEMPLATES
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#include "config.h"
#include "misc.h"
Fix Clang 5.0 "runtime error: addition of unsigned offset to 0xXXXX overflowed to 0xYYYY" (GH #549)
2017-12-16 23:18:53 +00:00
#include "stdcpp.h"
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
Fix ARM headers and Android compile (PR #896) * Test fix ARM headers This problem has been festering for some time. The header file includes are slightly different than the ISA options. Some platforms need an include, others don't. * Fix cryptest-android.sh and cryptest-ios.sh * Fix MSVC ARM32 and ARM64 compile * Split ARM32 and ARM64 recipes in GNUmakefile
2019-10-17 01:00:45 +00:00
#if (CRYPTOPP_ARM_NEON_HEADER)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
# include <arm_neon.h>
#endif
Fix ARM headers and Android compile (PR #896) * Test fix ARM headers This problem has been festering for some time. The header file includes are slightly different than the ISA options. Some platforms need an include, others don't. * Fix cryptest-android.sh and cryptest-ios.sh * Fix MSVC ARM32 and ARM64 compile * Split ARM32 and ARM64 recipes in GNUmakefile
2019-10-17 01:00:45 +00:00
#if (CRYPTOPP_ARM_ACLE_HEADER)
Add ARMv8.4 cpu feature detection support (GH #685) (#687) This PR adds ARMv8.4 cpu feature detection support. Previously we only needed ARMv8.1 and things were much easier. For example, ARMv8.1 `__ARM_FEATURE_CRYPTO` meant PMULL, AES, SHA-1 and SHA-256 were available. ARMv8.4 `__ARM_FEATURE_CRYPTO` means PMULL, AES, SHA-1, SHA-256, SHA-512, SHA-3, SM3 and SM4 are available. We still use the same pattern as before. We make something available based on compiler version and/or preprocessor macros. But this time around we had to tighten things up a bit to ensure ARMv8.4 did not cross-pollinate down into ARMv8.1. ARMv8.4 is largely untested at the moment. There is no hardware in the field and CI lacks QEMU with the relevant patches/support. We will probably have to revisit some of this stuff in the future. Since this update applies to ARM gadgets we took the time to expand Android and iOS testing on Travis. Travis now tests more platforms, and includes Autotools and CMake builds, too.
2018-07-15 12:35:14 +00:00
# include <stdint.h>
# include <arm_acle.h>
#endif
Fix build with Embarcadero C++Builder 10.2.3 (#696) Fix two compilation errors encountered with C++Builder (Starter Edition): - In `cpu.cpp`, 0ccdc197b introduced a dependency on `_xgetbv()` from `<immintrin.h>` that doesn't exist on C++Builder. Enlist it for the workaround, similar to SunCC in 692ed2a2b. - In `adv-simd.h`, `<pmmintrin.h>` is being #included under the `CRYPTOPP_SSE2_INTRIN_AVAILABLE` macro. This header, [which apparently provides SSE3 intrinsics](https://stackoverflow.com/a/11228864/1433768), is not shipped with C++Builder. (This section of code was recently downgraded from a SSSE3 to a SSE2 block in 09c8ae28, followed by moving away from `<immintrin.h>` in bc8da71a, followed by reintroducing the SSSE3 check in d1e646a5.) Split the SSE2 and SSSE3 cases such that `<pmmintrin.h>` is not #included for SSE2. This seems safe to do, because some `git grep` analysis shows that: - `adv-simd.h` is not #included by any other header, but only directly #included by some `.cpp` files. - Among those `.cpp` files, only `sm4-simd.cpp` has a `CRYPTOPP_SSE2_INTRIN_AVAILABLE` preprocessor block, and there it again includes the other two headers (`<emmintrin.h>` and `<xmmintrin.h>`). NOTE: I was compiling via the IDE after [setting up a project file](https://github.com/tanzislam/cryptopals/wiki/Importing-into-Embarcadero-C%E2%94%BC%E2%94%BCBuilder-Starter-10.2#using-the-crypto-library). My compilation command was effectively: ``` bcc32c.exe -DCRYPTOPP_NO_CXX11 -DCRYPTOPP_DISABLE_SSSE3 -D__SSE2__ -D__SSE__ -D__MMX__ ```
2018-08-05 02:54:36 +00:00
#if (CRYPTOPP_SSE2_INTRIN_AVAILABLE)
# include <emmintrin.h>
# include <xmmintrin.h>
#endif
Fix SunStudio 12.6 compile on i386
2018-07-16 13:37:08 +00:00
// SunCC needs CRYPTOPP_SSSE3_AVAILABLE, too
Fix build with Embarcadero C++Builder 10.2.3 (#696) Fix two compilation errors encountered with C++Builder (Starter Edition): - In `cpu.cpp`, 0ccdc197b introduced a dependency on `_xgetbv()` from `<immintrin.h>` that doesn't exist on C++Builder. Enlist it for the workaround, similar to SunCC in 692ed2a2b. - In `adv-simd.h`, `<pmmintrin.h>` is being #included under the `CRYPTOPP_SSE2_INTRIN_AVAILABLE` macro. This header, [which apparently provides SSE3 intrinsics](https://stackoverflow.com/a/11228864/1433768), is not shipped with C++Builder. (This section of code was recently downgraded from a SSSE3 to a SSE2 block in 09c8ae28, followed by moving away from `<immintrin.h>` in bc8da71a, followed by reintroducing the SSSE3 check in d1e646a5.) Split the SSE2 and SSSE3 cases such that `<pmmintrin.h>` is not #included for SSE2. This seems safe to do, because some `git grep` analysis shows that: - `adv-simd.h` is not #included by any other header, but only directly #included by some `.cpp` files. - Among those `.cpp` files, only `sm4-simd.cpp` has a `CRYPTOPP_SSE2_INTRIN_AVAILABLE` preprocessor block, and there it again includes the other two headers (`<emmintrin.h>` and `<xmmintrin.h>`). NOTE: I was compiling via the IDE after [setting up a project file](https://github.com/tanzislam/cryptopals/wiki/Importing-into-Embarcadero-C%E2%94%BC%E2%94%BCBuilder-Starter-10.2#using-the-crypto-library). My compilation command was effectively: ``` bcc32c.exe -DCRYPTOPP_NO_CXX11 -DCRYPTOPP_DISABLE_SSSE3 -D__SSE2__ -D__SSE__ -D__MMX__ ```
2018-08-05 02:54:36 +00:00
#if (CRYPTOPP_SSSE3_AVAILABLE)
Fix early Fedora compiles
2018-07-06 05:14:28 +00:00
# include <emmintrin.h>
# include <pmmintrin.h>
Use inline for LEA_Encryption and LEA_Decryption
2018-06-23 16:58:55 +00:00
# include <xmmintrin.h>
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#endif
Use __ALTIVEC__ preprocessor macro for PowerPC
2018-11-17 06:49:48 +00:00
#if defined(__ALTIVEC__)
Rename files with dashes to underscores (GH #736) Also see https://groups.google.com/forum/#!topic/cryptopp-users/HBz-6gZZFOA on the mailing list
2018-11-10 13:00:14 +00:00
# include "ppc_simd.h"
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
#endif
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
// ************************ All block ciphers *********************** //
ANONYMOUS_NAMESPACE_BEGIN
using CryptoPP::BlockTransformation;
Fix semicolons in adv_simd.h Many, that sed went sideways...
2019-10-13 17:39:34 +00:00
CRYPTOPP_CONSTANT(BT_XorInput = BlockTransformation::BT_XorInput);
CRYPTOPP_CONSTANT(BT_AllowParallel = BlockTransformation::BT_AllowParallel);
CRYPTOPP_CONSTANT(BT_InBlockIsCounter = BlockTransformation::BT_InBlockIsCounter);
CRYPTOPP_CONSTANT(BT_ReverseDirection = BlockTransformation::BT_ReverseDirection);
CRYPTOPP_CONSTANT(BT_DontIncrementInOutPointers = BlockTransformation::BT_DontIncrementInOutPointers);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
ANONYMOUS_NAMESPACE_END
// *************************** ARM NEON ************************** //
Add CRYPTOPP_ARM_ASIMD_AVAILABLE define for Aarch64
2020-07-29 03:06:34 +00:00
#if (CRYPTOPP_ARM_NEON_AVAILABLE) || (CRYPTOPP_ARM_ASIMD_AVAILABLE) || \
defined(CRYPTOPP_DOXYGEN_PROCESSING)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
NAMESPACE_BEGIN(CryptoPP)
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \brief AdvancedProcessBlocks for 1 and 6 blocks
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
/// \tparam F1 function to process 1 128-bit block
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \tparam F6 function to process 6 128-bit blocks
/// \tparam W word type of the subkey table
Fix AdvancedProcessBlocks128_6x1_NEON template name
2018-06-23 16:35:06 +00:00
/// \details AdvancedProcessBlocks128_6x1_NEON processes 6 and 2 NEON SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \details The subkey type is usually word32 or word64. F1 and F6 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
template <typename F1, typename F6, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_6x1_NEON(F1 func1, F6 func6,
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
const W *subKeys, size_t rounds, const byte *inBlocks,
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Use carryless multiplies for NIST b233 and k233 curves (GH #783, PR #784) Use carryless multiplies for NIST b233 and k233 curves.
2019-01-16 05:02:04 +00:00
const unsigned int w_one[] = {0, 0<<24, 0, 1<<24};
const uint32x4_t s_one = vld1q_u32(w_one);
Fix ODR violation in AdvancedProcessBlocks_{ARCH} (GH #585) The ALTIVEC function required an inline declaration. Lack of inline caused the self test failure. Two NEON functions needed the same. We also cleaned up constants in unnamed namespaces
2018-02-20 18:17:05 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
const size_t blockSize = 16;
// const size_t neonBlockSize = 16;
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
if (flags & BT_ReverseDirection)
{
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 6*blockSize)
{
uint64x2_t block0, block1, block2, block3, block4, block5;
if (flags & BT_InBlockIsCounter)
{
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
const uint64x2_t one = vreinterpretq_u64_u32(s_one);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
block1 = vaddq_u64(block0, one);
block2 = vaddq_u64(block1, one);
block3 = vaddq_u64(block2, one);
block4 = vaddq_u64(block3, one);
block5 = vaddq_u64(block4, one);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(const_cast<byte*>(inBlocks),
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
vreinterpretq_u8_u64(vaddq_u64(block5, one)));
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
else
{
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block4 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block5 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = veorq_u64(block2, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = veorq_u64(block3, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block4 = veorq_u64(block4, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block5 = veorq_u64(block5, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
func6(block0, block1, block2, block3, block4, block5, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = veorq_u64(block2, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = veorq_u64(block3, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block4 = veorq_u64(block4, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block5 = veorq_u64(block5, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block0));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block1));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block2));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block3));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block4));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block5));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
length -= 6*blockSize;
}
}
while (length >= blockSize)
{
uint64x2_t block;
block = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block = veorq_u64(block, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func1(block, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block = veorq_u64(block, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
length -= blockSize;
}
return length;
}
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
/// \brief AdvancedProcessBlocks for 1 and 4 blocks
/// \tparam F1 function to process 1 128-bit block
/// \tparam F4 function to process 4 128-bit blocks
/// \tparam W word type of the subkey table
Update documentation
2018-06-23 16:27:25 +00:00
/// \details AdvancedProcessBlocks128_4x1_NEON processes 4 and 1 NEON SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time.
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
/// \details The subkey type is usually word32 or word64. V is the vector type and it is
Update documentation
2019-10-29 14:33:39 +00:00
/// usually uint32x4_t or uint32x4_t. F1, F4, and W must use the same word and
/// vector type.
Fix AdvancedProcessBlocks_{MxN}_NEON compile with MSVC compiler (GH #776)
2019-01-04 00:49:00 +00:00
template <typename F1, typename F4, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_4x1_NEON(F1 func1, F4 func4,
Fix AdvancedProcessBlocks_{MxN}_NEON compile with MSVC compiler (GH #776)
2019-01-04 00:49:00 +00:00
const W *subKeys, size_t rounds, const byte *inBlocks,
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Use carryless multiplies for NIST b233 and k233 curves (GH #783, PR #784) Use carryless multiplies for NIST b233 and k233 curves.
2019-01-16 05:02:04 +00:00
const unsigned int w_one[] = {0, 0<<24, 0, 1<<24};
const uint32x4_t s_one = vld1q_u32(w_one);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
const size_t blockSize = 16;
// const size_t neonBlockSize = 16;
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
if (flags & BT_ReverseDirection)
{
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 4*blockSize)
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
uint32x4_t block0, block1, block2, block3;
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
if (flags & BT_InBlockIsCounter)
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
const uint32x4_t one = s_one;
block0 = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Use 64-bit add for AdvancedProcessBlocks128_4x1_NEON
2019-01-04 02:25:19 +00:00
block1 = vreinterpretq_u32_u64(vaddq_u64(vreinterpretq_u64_u32(block0), vreinterpretq_u64_u32(one)));
block2 = vreinterpretq_u32_u64(vaddq_u64(vreinterpretq_u64_u32(block1), vreinterpretq_u64_u32(one)));
block3 = vreinterpretq_u32_u64(vaddq_u64(vreinterpretq_u64_u32(block2), vreinterpretq_u64_u32(one)));
vst1q_u8(const_cast<byte*>(inBlocks), vreinterpretq_u8_u64(vaddq_u64(
vreinterpretq_u64_u32(block3), vreinterpretq_u64_u32(one))));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
}
else
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block0 = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block1 = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block2 = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block3 = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
}
if (xorInput)
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block0 = veorq_u32(block0, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block1 = veorq_u32(block1, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block2 = veorq_u32(block2, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block3 = veorq_u32(block3, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
}
Fix AdvancedProcessBlocks_{MxN}_NEON compile with MSVC compiler (GH #776)
2019-01-04 00:49:00 +00:00
func4(block0, block1, block2, block3, subKeys, static_cast<unsigned int>(rounds));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
if (xorOutput)
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block0 = veorq_u32(block0, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block1 = veorq_u32(block1, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block2 = veorq_u32(block2, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block3 = veorq_u32(block3, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
}
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u32(block0));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u32(block1));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u32(block2));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u32(block3));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
length -= 4*blockSize;
}
Remove GCC_NO_UBSAN and double casts
2018-07-01 05:23:35 +00:00
}
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
while (length >= blockSize)
{
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
uint32x4_t block = vreinterpretq_u32_u8(vld1q_u8(inBlocks));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
if (xorInput)
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block = veorq_u32(block, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
Fix AdvancedProcessBlocks_{MxN}_NEON compile with MSVC compiler (GH #776)
2019-01-04 00:49:00 +00:00
func1(block, subKeys, static_cast<unsigned int>(rounds));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
if (xorOutput)
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
block = veorq_u32(block, vreinterpretq_u32_u8(vld1q_u8(xorBlocks)));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
Fix self test NEON compile with MSVC compiler (GH #776)
2019-01-04 02:00:22 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u32(block));
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add LEA-128 NEON and ARMv8 implementation (GH #669) LEA-128(128) from 35.6 cpb to 14.11 cpb on a LeMaker HiKey dev-board. LEA-128 from 12.60 cpb to 11.89 cpb on AMD Opteron 1100.
2018-06-23 07:54:51 +00:00
length -= blockSize;
}
return length;
}
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \brief AdvancedProcessBlocks for 2 and 6 blocks
/// \tparam F2 function to process 2 128-bit blocks
/// \tparam F6 function to process 6 128-bit blocks
/// \tparam W word type of the subkey table
/// \details AdvancedProcessBlocks128_6x2_NEON processes 6 and 2 NEON SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time. For a single block the template uses F2 with a zero block.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \details The subkey type is usually word32 or word64. F2 and F6 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
template <typename F2, typename F6, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_6x2_NEON(F2 func2, F6 func6,
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
const W *subKeys, size_t rounds, const byte *inBlocks,
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Use carryless multiplies for NIST b233 and k233 curves (GH #783, PR #784) Use carryless multiplies for NIST b233 and k233 curves.
2019-01-16 05:02:04 +00:00
const unsigned int w_one[] = {0, 0<<24, 0, 1<<24};
const uint32x4_t s_one = vld1q_u32(w_one);
Fix ODR violation in AdvancedProcessBlocks_{ARCH} (GH #585) The ALTIVEC function required an inline declaration. Lack of inline caused the self test failure. Two NEON functions needed the same. We also cleaned up constants in unnamed namespaces
2018-02-20 18:17:05 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
const size_t blockSize = 16;
// const size_t neonBlockSize = 16;
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
if (flags & BT_ReverseDirection)
{
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 6*blockSize)
{
uint64x2_t block0, block1, block2, block3, block4, block5;
if (flags & BT_InBlockIsCounter)
{
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
const uint64x2_t one = vreinterpretq_u64_u32(s_one);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
block1 = vaddq_u64(block0, one);
block2 = vaddq_u64(block1, one);
block3 = vaddq_u64(block2, one);
block4 = vaddq_u64(block3, one);
block5 = vaddq_u64(block4, one);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(const_cast<byte*>(inBlocks),
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
vreinterpretq_u8_u64(vaddq_u64(block5, one)));
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
else
{
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block2 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block3 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block4 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block5 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block2 = veorq_u64(block2, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block3 = veorq_u64(block3, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block4 = veorq_u64(block4, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block5 = veorq_u64(block5, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
func6(block0, block1, block2, block3, block4, block5, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block2 = veorq_u64(block2, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block3 = veorq_u64(block3, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block4 = veorq_u64(block4, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block5 = veorq_u64(block5, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block0));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block1));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block2));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block3));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block4));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block5));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
length -= 6*blockSize;
}
while (length >= 2*blockSize)
{
uint64x2_t block0, block1;
if (flags & BT_InBlockIsCounter)
{
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
const uint64x2_t one = vreinterpretq_u64_u32(s_one);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
block1 = vaddq_u64(block0, one);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(const_cast<byte*>(inBlocks),
Cleanup adv-simd.h for ARM
2018-08-12 23:51:50 +00:00
vreinterpretq_u8_u64(vaddq_u64(block1, one)));
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
else
{
block0 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
func2(block0, block1, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = veorq_u64(block0, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = veorq_u64(block1, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block0));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block1));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
length -= 2*blockSize;
}
}
while (length >= blockSize)
{
uint64x2_t block, zero = {0,0};
block = vreinterpretq_u64_u8(vld1q_u8(inBlocks));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block = veorq_u64(block, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func2(block, zero, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block = veorq_u64(block, vreinterpretq_u64_u8(vld1q_u8(xorBlocks)));
vst1q_u8(outBlocks, vreinterpretq_u8_u64(block));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
length -= blockSize;
}
return length;
}
Fix unaligned load for _mm_loaddup_pd with GCC and UBsan This function was missed earlier. Unfortunately, it does not squash all of the unaligned load findings. I'm pretty sure it is a GCC problem
2017-12-28 06:16:17 +00:00
NAMESPACE_END // CryptoPP
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#endif // CRYPTOPP_ARM_NEON_AVAILABLE
// *************************** Intel SSE ************************** //
Update documentation
2019-10-29 14:16:11 +00:00
#if defined(CRYPTOPP_SSSE3_AVAILABLE) || defined(CRYPTOPP_DOXYGEN_PROCESSING)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
Update documentation
2019-10-29 14:33:39 +00:00
#if defined(CRYPTOPP_DOXYGEN_PROCESSING)
/// \brief SunCC workaround
/// \details SunCC loses the const on AES_Enc_Block and AES_Dec_Block
/// \sa <A HREF="http://github.com/weidai11/cryptopp/issues/224">Issue
/// 224, SunCC and failed compile for rijndael.cpp</A>
# define MAYBE_CONST const
/// \brief SunCC workaround
/// \details SunCC loses the const on AES_Enc_Block and AES_Dec_Block
/// \sa <A HREF="http://github.com/weidai11/cryptopp/issues/224">Issue
/// 224, SunCC and failed compile for rijndael.cpp</A>
# define MAYBE_UNCONST_CAST(T, x) (x)
#elif (__SUNPRO_CC >= 0x5130)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
# define MAYBE_CONST
# define MAYBE_UNCONST_CAST(T, x) const_cast<MAYBE_CONST T>(x)
#else
# define MAYBE_CONST const
# define MAYBE_UNCONST_CAST(T, x) (x)
#endif
Update documentation
2019-10-29 14:33:39 +00:00
#if defined(CRYPTOPP_DOXYGEN_PROCESSING)
/// \brief Clang workaround
/// \details Clang issues spurious alignment warnings
/// \sa <A HREF="http://bugs.llvm.org/show_bug.cgi?id=20670">Issue
/// 20670, _mm_loadu_si128 parameter has wrong type</A>
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
# define M128_CAST(x) ((__m128i *)(void *)(x))
Update documentation
2019-10-29 14:33:39 +00:00
/// \brief Clang workaround
/// \details Clang issues spurious alignment warnings
/// \sa <A HREF="http://bugs.llvm.org/show_bug.cgi?id=20670">Issue
/// 20670, _mm_loadu_si128 parameter has wrong type</A>
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
# define CONST_M128_CAST(x) ((const __m128i *)(const void *)(x))
Update documentation
2019-10-29 14:33:39 +00:00
#else
# ifndef M128_CAST
# define M128_CAST(x) ((__m128i *)(void *)(x))
# endif
# ifndef CONST_M128_CAST
# define CONST_M128_CAST(x) ((const __m128i *)(const void *)(x))
# endif
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#endif
NAMESPACE_BEGIN(CryptoPP)
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
/// \brief AdvancedProcessBlocks for 2 and 6 blocks
/// \tparam F2 function to process 2 128-bit blocks
/// \tparam F6 function to process 6 128-bit blocks
Update documentation
2018-07-01 07:29:12 +00:00
/// \tparam W word type of the subkey table
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
/// \details AdvancedProcessBlocks128_6x2_SSE processes 6 and 2 SSE SIMD words
/// at a time. For a single block the template uses F2 with a zero block.
/// \details The subkey type is usually word32 or word64. F2 and F6 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
template <typename F2, typename F6, typename W>
inline size_t AdvancedProcessBlocks128_6x2_SSE(F2 func2, F6 func6,
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
MAYBE_CONST W *subKeys, size_t rounds, const byte *inBlocks,
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
CRYPTOPP_ASSERT(length >= 16);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
const size_t blockSize = 16;
// const size_t xmmBlockSize = 16;
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
if (flags & BT_ReverseDirection)
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
while (length >= 6*blockSize)
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
__m128i block0, block1, block2, block3, block4, block5;
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
if (flags & BT_InBlockIsCounter)
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
// Increment of 1 in big-endian compatible with the ctr byte array.
Cleanup adv-simd.h for x86
2018-08-12 23:04:14 +00:00
const __m128i s_one = _mm_set_epi32(1<<24, 0, 0, 0);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
block1 = _mm_add_epi32(block0, s_one);
block2 = _mm_add_epi32(block1, s_one);
block3 = _mm_add_epi32(block2, s_one);
block4 = _mm_add_epi32(block3, s_one);
block5 = _mm_add_epi32(block4, s_one);
_mm_storeu_si128(M128_CAST(inBlocks), _mm_add_epi32(block5, s_one));
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
else
{
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
block1 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
block2 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
block3 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
block4 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
block5 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
if (xorInput)
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
block2 = _mm_xor_si128(block2, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block3 = _mm_xor_si128(block3, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block4 = _mm_xor_si128(block4, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block5 = _mm_xor_si128(block5, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
func6(block0, block1, block2, block3, block4, block5, subKeys, static_cast<unsigned int>(rounds));
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
if (xorOutput)
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
block2 = _mm_xor_si128(block2, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block3 = _mm_xor_si128(block3, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block4 = _mm_xor_si128(block4, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
block5 = _mm_xor_si128(block5, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
_mm_storeu_si128(M128_CAST(outBlocks), block0);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block1);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block2);
outBlocks = PtrAdd(outBlocks, outIncrement);
_mm_storeu_si128(M128_CAST(outBlocks), block3);
outBlocks = PtrAdd(outBlocks, outIncrement);
_mm_storeu_si128(M128_CAST(outBlocks), block4);
outBlocks = PtrAdd(outBlocks, outIncrement);
_mm_storeu_si128(M128_CAST(outBlocks), block5);
outBlocks = PtrAdd(outBlocks, outIncrement);
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
length -= 6*blockSize;
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
while (length >= 2*blockSize)
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
__m128i block0, block1;
if (flags & BT_InBlockIsCounter)
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
// Increment of 1 in big-endian compatible with the ctr byte array.
const __m128i s_one = _mm_set_epi32(1<<24, 0, 0, 0);
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
block1 = _mm_add_epi32(block0, s_one);
_mm_storeu_si128(M128_CAST(inBlocks), _mm_add_epi32(block1, s_one));
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
}
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
else
Add CHAM64 SSSE3 implementation (PR #670) CHAM64 from 20 cpb to 14 cpb on modern iCore. CHAM64 from 90 cpb to 18 cpb antique Core2 Duo
2018-06-21 04:37:10 +00:00
{
Remove 64-bit AdvancedProcessBlocks (GH #945)
2020-07-07 19:22:09 +00:00
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
block1 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
func2(block0, block1, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
}
_mm_storeu_si128(M128_CAST(outBlocks), block0);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block1);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
length -= 2*blockSize;
}
}
while (length >= blockSize)
{
__m128i block, zero = _mm_setzero_si128();
block = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block = _mm_xor_si128(block, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func2(block, zero, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
block = _mm_xor_si128(block, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
_mm_storeu_si128(M128_CAST(outBlocks), block);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
length -= blockSize;
}
return length;
}
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \brief AdvancedProcessBlocks for 1 and 4 blocks
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
/// \tparam F1 function to process 1 128-bit block
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \tparam F4 function to process 4 128-bit blocks
/// \tparam W word type of the subkey table
/// \details AdvancedProcessBlocks128_4x1_SSE processes 4 and 1 SSE SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time.
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
/// \details The subkey type is usually word32 or word64. F1 and F4 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
template <typename F1, typename F4, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_4x1_SSE(F1 func1, F4 func4,
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
MAYBE_CONST W *subKeys, size_t rounds, const byte *inBlocks,
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
const size_t blockSize = 16;
// const size_t xmmBlockSize = 16;
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
if (flags & BT_ReverseDirection)
{
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 4*blockSize)
{
__m128i block0, block1, block2, block3;
if (flags & BT_InBlockIsCounter)
{
Cleanup adv-simd.h for x86
2018-08-12 23:04:14 +00:00
// Increment of 1 in big-endian compatible with the ctr byte array.
const __m128i s_one = _mm_set_epi32(1<<24, 0, 0, 0);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Cleanup adv-simd.h for x86
2018-08-12 23:04:14 +00:00
block1 = _mm_add_epi32(block0, s_one);
block2 = _mm_add_epi32(block1, s_one);
block3 = _mm_add_epi32(block2, s_one);
_mm_storeu_si128(M128_CAST(inBlocks), _mm_add_epi32(block3, s_one));
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
else
{
block0 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = _mm_xor_si128(block2, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = _mm_xor_si128(block3, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
func4(block0, block1, block2, block3, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
{
block0 = _mm_xor_si128(block0, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block1 = _mm_xor_si128(block1, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block2 = _mm_xor_si128(block2, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block3 = _mm_xor_si128(block3, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
}
_mm_storeu_si128(M128_CAST(outBlocks), block0);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block1);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block2);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
_mm_storeu_si128(M128_CAST(outBlocks), block3);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
length -= 4*blockSize;
}
}
while (length >= blockSize)
{
__m128i block = _mm_loadu_si128(CONST_M128_CAST(inBlocks));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block = _mm_xor_si128(block, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func1(block, subKeys, static_cast<unsigned int>(rounds));
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
block = _mm_xor_si128(block, _mm_loadu_si128(CONST_M128_CAST(xorBlocks)));
_mm_storeu_si128(M128_CAST(outBlocks), block);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Refactor rijndael-simd.cpp and simon.simd.cpp to use adv-simd.h
2017-12-10 16:09:50 +00:00
length -= blockSize;
}
return length;
}
Fix unaligned load for _mm_loaddup_pd with GCC and UBsan This function was missed earlier. Unfortunately, it does not squash all of the unaligned load findings. I'm pretty sure it is a GCC problem
2017-12-28 06:16:17 +00:00
NAMESPACE_END // CryptoPP
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#endif // CRYPTOPP_SSSE3_AVAILABLE
Update documentation
2019-10-29 14:16:11 +00:00
// ************************** Altivec/Power 4 ************************** //
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Update documentation
2019-10-29 14:16:11 +00:00
#if defined(__ALTIVEC__) || defined(CRYPTOPP_DOXYGEN_PROCESSING)
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
NAMESPACE_BEGIN(CryptoPP)
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
/// \brief AdvancedProcessBlocks for 1 and 4 blocks
/// \tparam F1 function to process 1 128-bit block
/// \tparam F4 function to process 4 128-bit blocks
/// \tparam W word type of the subkey table
/// \details AdvancedProcessBlocks128_4x1_ALTIVEC processes 4 and 1 Altivec SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time.
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
/// \details The subkey type is usually word32 or word64. F1 and F4 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
template <typename F1, typename F4, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_4x1_ALTIVEC(F1 func1, F4 func4,
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
const W *subKeys, size_t rounds, const byte *inBlocks,
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Sync CRYPTOPP_{BIG|LITTLE}_ENDIAN with Autotools Autotools sets up its config.h file with the '#define XXX 0' or '#define XXX 1' pattern. This check-in makes the sources Autotools aware. We need to verify CMake does the same
2018-10-28 08:24:22 +00:00
#if (CRYPTOPP_LITTLE_ENDIAN)
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
const uint32x4_p s_one = {1,0,0,0};
#else
const uint32x4_p s_one = {0,0,0,1};
#endif
const size_t blockSize = 16;
Use simdBlockSize for Altivec. The VSX unit is Power7
2019-10-29 14:18:36 +00:00
// const size_t simdBlockSize = 16;
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
if (flags & BT_ReverseDirection)
{
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 4*blockSize)
{
uint32x4_p block0, block1, block2, block3;
if (flags & BT_InBlockIsCounter)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecLoadBE(inBlocks);
block1 = VecAdd(block0, s_one);
block2 = VecAdd(block1, s_one);
block3 = VecAdd(block2, s_one);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
// Hack due to big-endian loads used by POWER8 (and maybe ARM-BE).
// CTR_ModePolicy::OperateKeystream is wired such that after
// returning from this function CTR_ModePolicy will detect wrap on
// on the last counter byte and increment the next to last byte.
// The problem is, with a big-endian load, inBlocks[15] is really
// located at index 15. The vector addition using a 32-bit element
// generates a carry into inBlocks[14] and then CTR_ModePolicy
// increments inBlocks[14] too.
Use PtrSub to subtract offset from pointer
2018-08-15 07:16:08 +00:00
const_cast<byte*>(inBlocks)[15] += 6;
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
}
else
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecLoadBE(inBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecLoadBE(inBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecLoadBE(inBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecLoadBE(inBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
}
if (xorInput)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecXor(block0, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecXor(block1, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecXor(block2, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecXor(block3, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
}
func4(block0, block1, block2, block3, subKeys, rounds);
if (xorOutput)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecXor(block0, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecXor(block1, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecXor(block2, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecXor(block3, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
}
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block0, outBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block1, outBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block2, outBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block3, outBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
length -= 4*blockSize;
}
}
while (length >= blockSize)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
uint32x4_p block = VecLoadBE(inBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
if (xorInput)
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block = VecXor(block, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func1(block, subKeys, rounds);
if (xorOutput)
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block = VecXor(block, VecLoadBE(xorBlocks));
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block, outBlocks);
Add AdvancedProcessBlocks128_4x1_ALTIVEC template
2018-08-13 10:38:30 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
length -= blockSize;
}
return length;
}
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \brief AdvancedProcessBlocks for 1 and 6 blocks
Add LEA-128 SSSE3 implementation (GH #669) LEA-128(128) from 6.73 cpb to 2.84 cpb on modern Core-i5 6400. LEA-128 from 10.12 cpb to 7.84 cpb antique Core2 Duo.
2018-06-22 20:26:27 +00:00
/// \tparam F1 function to process 1 128-bit block
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \tparam F6 function to process 6 128-bit blocks
/// \tparam W word type of the subkey table
/// \details AdvancedProcessBlocks128_6x1_ALTIVEC processes 6 and 1 Altivec SIMD words
Update documentation
2019-10-29 14:33:39 +00:00
/// at a time.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
/// \details The subkey type is usually word32 or word64. F1 and F6 must use the
Update documentation
2019-10-29 14:33:39 +00:00
/// same word type.
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
template <typename F1, typename F6, typename W>
Fix "explicit template specialization cannot have a storage class" on Aarch64 debug builds
2018-12-26 18:24:54 +00:00
inline size_t AdvancedProcessBlocks128_6x1_ALTIVEC(F1 func1, F6 func6,
Parameterize word type for subkeys in AdvancedProcessBlocks templates This was needed a while ago but we mostly side-stepped the issues with casts. CHAM64 uses a word16 type for subkeys and a cast won't fix it because we favor word32 for 64-bit block sizes.
2018-06-20 23:25:52 +00:00
const W *subKeys, size_t rounds, const byte *inBlocks,
Fix ODR violation in AdvancedProcessBlocks_{ARCH} (GH #585) The ALTIVEC function required an inline declaration. Lack of inline caused the self test failure. Two NEON functions needed the same. We also cleaned up constants in unnamed namespaces
2018-02-20 18:17:05 +00:00
const byte *xorBlocks, byte *outBlocks, size_t length, word32 flags)
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
{
CRYPTOPP_ASSERT(subKeys);
CRYPTOPP_ASSERT(inBlocks);
CRYPTOPP_ASSERT(outBlocks);
CRYPTOPP_ASSERT(length >= 16);
Sync CRYPTOPP_{BIG|LITTLE}_ENDIAN with Autotools Autotools sets up its config.h file with the '#define XXX 0' or '#define XXX 1' pattern. This check-in makes the sources Autotools aware. We need to verify CMake does the same
2018-10-28 08:24:22 +00:00
#if (CRYPTOPP_LITTLE_ENDIAN)
Fix ODR violation in AdvancedProcessBlocks_{ARCH} (GH #585) The ALTIVEC function required an inline declaration. Lack of inline caused the self test failure. Two NEON functions needed the same. We also cleaned up constants in unnamed namespaces
2018-02-20 18:17:05 +00:00
const uint32x4_p s_one = {1,0,0,0};
#else
const uint32x4_p s_one = {0,0,0,1};
#endif
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
const size_t blockSize = 16;
Use simdBlockSize for Altivec. The VSX unit is Power7
2019-10-29 14:18:36 +00:00
// const size_t simdBlockSize = 16;
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
size_t inIncrement = (flags & (BT_InBlockIsCounter|BT_DontIncrementInOutPointers)) ? 0 : blockSize;
size_t xorIncrement = (xorBlocks != NULLPTR) ? blockSize : 0;
size_t outIncrement = (flags & BT_DontIncrementInOutPointers) ? 0 : blockSize;
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
// Clang and Coverity are generating findings using xorBlocks as a flag.
const bool xorInput = (xorBlocks != NULLPTR) && (flags & BT_XorInput);
const bool xorOutput = (xorBlocks != NULLPTR) && !(flags & BT_XorInput);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
if (flags & BT_ReverseDirection)
{
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, length - blockSize);
xorBlocks = PtrAdd(xorBlocks, length - blockSize);
outBlocks = PtrAdd(outBlocks, length - blockSize);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
inIncrement = 0-inIncrement;
xorIncrement = 0-xorIncrement;
outIncrement = 0-outIncrement;
}
if (flags & BT_AllowParallel)
{
while (length >= 6*blockSize)
{
Fix carry bug in AdvancedProcessBlocks128_6x1_ALTIVEC
2018-08-13 05:44:23 +00:00
uint32x4_p block0, block1, block2, block3, block4, block5;
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
if (flags & BT_InBlockIsCounter)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecLoadBE(inBlocks);
block1 = VecAdd(block0, s_one);
block2 = VecAdd(block1, s_one);
block3 = VecAdd(block2, s_one);
block4 = VecAdd(block3, s_one);
block5 = VecAdd(block4, s_one);
Fix carry bug in AdvancedProcessBlocks128_6x1_ALTIVEC
2018-08-13 05:44:23 +00:00
// Hack due to big-endian loads used by POWER8 (and maybe ARM-BE).
// CTR_ModePolicy::OperateKeystream is wired such that after
Update comments
2018-08-13 05:51:01 +00:00
// returning from this function CTR_ModePolicy will detect wrap on
// on the last counter byte and increment the next to last byte.
// The problem is, with a big-endian load, inBlocks[15] is really
// located at index 15. The vector addition using a 32-bit element
// generates a carry into inBlocks[14] and then CTR_ModePolicy
// increments inBlocks[14] too.
Fix carry bug in AdvancedProcessBlocks128_6x1_ALTIVEC
2018-08-13 05:44:23 +00:00
//
// To find this bug we needed a test case with a ctr of 0xNN...FA.
// The last octet is 0xFA and adding 6 creates the wrap to trigger
// the issue. If the last octet was 0xFC then 4 would trigger it.
// We dumb-lucked into the test with SPECK-128. The test case of
// interest is the one with IV 348ECA9766C09F04 826520DE47A212FA.
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
uint8x16_p temp = VecAdd((uint8x16_p)block5, (uint8x16_p)s_one);
VecStoreBE(temp, const_cast<byte*>(inBlocks));
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
}
else
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block4 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block5 = VecLoadBE(inBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
}
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecXor(block0, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecXor(block1, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecXor(block2, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecXor(block3, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block4 = VecXor(block4, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block5 = VecXor(block5, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
}
func6(block0, block1, block2, block3, block4, block5, subKeys, rounds);
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block0 = VecXor(block0, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block1 = VecXor(block1, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block2 = VecXor(block2, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block3 = VecXor(block3, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block4 = VecXor(block4, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block5 = VecXor(block5, VecLoadBE(xorBlocks));
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
}
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block0, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block1, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block2, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block3, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block4, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block5, outBlocks);
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
outBlocks = PtrAdd(outBlocks, outIncrement);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
length -= 6*blockSize;
}
}
while (length >= blockSize)
{
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
uint32x4_p block = VecLoadBE(inBlocks);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorInput)
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block = VecXor(block, VecLoadBE(xorBlocks));
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
if (flags & BT_InBlockIsCounter)
const_cast<byte *>(inBlocks)[15]++;
func1(block, subKeys, rounds);
Add xorInput and xorOutput flags to adv-simd classes Analysis tools are generating findings when the pointer xorBlocks is used as the flag. The other missing piece is, xorBlocks is never NULL when either BT_XorInput or BT_XorOuput. But we don't know how to train the analyzers with the information, so we make it explicit with the boolean flags xorInput and xorOutput. Switching to the explicit flags costs us about 0.01 cpb on a modern Intel Core processor. In the typical case 0.01 is negligible.
2018-01-24 17:06:15 +00:00
if (xorOutput)
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
block = VecXor(block, VecLoadBE(xorBlocks));
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Rename PPC vector functions from VectorFunc to VecFunc
2018-11-15 20:17:49 +00:00
VecStoreBE(block, outBlocks);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Add PtrAdd and PtrSub helper functions This helps contain UB on pointer subtraction by ensuring a ptrdiff_t is used. The code is a little uglier but it is also more portable.
2018-07-10 09:00:02 +00:00
inBlocks = PtrAdd(inBlocks, inIncrement);
outBlocks = PtrAdd(outBlocks, outIncrement);
xorBlocks = PtrAdd(xorBlocks, xorIncrement);
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
length -= blockSize;
}
return length;
}
NAMESPACE_END // CryptoPP
Use __ALTIVEC__ preprocessor macro for PowerPC
2018-11-17 06:49:48 +00:00
#endif // __ALTIVEC__
Move Altivec AdvancedProcessBlocks into adv-simd.h
2018-01-02 12:08:13 +00:00
Refactor <cipher>_AdvancedProcessBlocks_<arch> into adv-simd.h This also fixes the SPECK64 bug where CTR mode self tests fail. It was an odd failure because it only affected 64-bit SPECK. SIMON was fine and it used nearly the same code. We tracked it down through trial and error to the table based rotates.
2017-12-10 02:04:25 +00:00
#endif // CRYPTOPP_ADVANCED_SIMD_TEMPLATES
Reference in New Issue Copy Permalink