OXYGENHUB
# c8d9059e4e70 oxygen / vendor / crc-fast / .kiro / specs / crc16-hardware-acceleration / design.md
13.9 KiB · Markdown · last changed Strata: the cloud stores send through ureq, so every release target links 1 month ago renderedsource blame history raw
2cd40ea1 Andy McCall ↶ 1
# Design Document: CRC-16 Hardware Acceleration
2
3
## Overview
4
5
This design extends the crc-fast library to support hardware-accelerated CRC-16 computation using the same PCLMULQDQ/PMULL-based approach already implemented for CRC-32 and CRC-64. The key insight from the reference implementations is that CRC-16 computation can be performed by scaling the 16-bit values to 32-bit space, using the existing CRC-32 algorithm infrastructure, and then scaling the result back to 16 bits.
6
7
The implementation follows a two-phase approach:
8
1. **Phase 1**: Extend the key generator (`generate.rs`) to produce correct CRC-16 folding keys
9
2. **Phase 2**: Extend the algorithm module to handle width=16 by leveraging the CRC-32 code path with appropriate scaling
10
11
## Architecture
12
13
### High-Level Design
14
15
```
16
┌─────────────────────────────────────────────────────────────────────┐
17
│                         Public API                                   │
18
│  checksum(), Digest, checksum_with_params(), CrcParams::new()       │
19
└─────────────────────────────────────────────────────────────────────┘
20
                                    │
21
                                    ▼
22
┌─────────────────────────────────────────────────────────────────────┐
23
│                      Width Dispatcher                                │
24
│  arch/mod.rs: routes to Width16, Width32, or Width64                │
25
└─────────────────────────────────────────────────────────────────────┘
26
                                    │
27
                    ┌───────────────┼───────────────┐
28
                    ▼               ▼               ▼
29
              ┌──────────┐   ┌──────────┐   ┌──────────┐
30
              │ Width16  │   │ Width32  │   │ Width64  │
31
              │ (NEW)    │   │(existing)│   │(existing)│
32
              └──────────┘   └──────────┘   └──────────┘
33
                    │               │               │
34
                    └───────────────┼───────────────┘
35
                                    ▼
36
┌─────────────────────────────────────────────────────────────────────┐
37
│                    SIMD Algorithm Core                               │
38
│  algorithm.rs: fold-by-8, Barrett reduction                         │
39
└─────────────────────────────────────────────────────────────────────┘
40
```
41
42
### CRC-16 Scaling Strategy
43
44
The reference assembly implementations reveal that CRC-16 uses the same PCLMULQDQ algorithm as CRC-32, but with values scaled to 32-bit space:
45
46
**Forward (non-reflected) CRC-16:**
47
1. Scale initial CRC: `initial_crc << 16` (shift left 16 bits)
48
2. Process using CRC-32 algorithm with CRC-16 polynomial scaled to 32 bits
49
3. Scale result: `result >> 16` (shift right 16 bits)
50
51
**Reflected CRC-16:**
52
1. Initial CRC used directly (no scaling needed for reflected)
53
2. Process using CRC-32 algorithm with reflected CRC-16 polynomial
54
3. Result extracted from appropriate register position
55
56
## Components and Interfaces
57
58
### 1. Key Generator Extension (`src/generate.rs`)
59
60
The key generator must be extended to handle width=16. CRC-16 uses the same exponents as CRC-32 (32*N distances) because the folding algorithm operates on 128-bit SIMD registers regardless of CRC width.
61
62
```rust
63
/// Exponents for CRC-16 key generation (same as CRC-32)
64
const CRC16_EXPONENTS: [u64; 23] = CRC32_EXPONENTS;
65
66
/// Generates folding keys for CRC-16
67
pub fn keys(width: u8, poly: u64, reflected: bool) -> [u64; 23] {
68
    let exponents = match width {
69
        16 => CRC16_EXPONENTS,  // NEW
70
        32 => CRC32_EXPONENTS,
71
        64 => CRC64_EXPONENTS,
72
        _ => panic!("Unsupported width: {width}"),
73
    };
74
    // ... rest of implementation
75
}
76
```
77
78
**Key Generation Functions:**
79
80
```rust
81
/// Computes a CRC-16 folding key for a given bit distance
82
fn crc16_key(exponent: u64, reflected: bool, polynomial: u64) -> u64 {
83
    // CRC-16 uses 17-bit polynomial (16 bits + implicit leading 1)
84
    // Algorithm similar to CRC-32 but with different bit positions
85
    if exponent < 32 {
86
        return 0;
87
    }
88
    
89
    let mut n: u64 = 0x080000000;  // Start at x^32 (same as CRC-32)
90
    let e = exponent - 31;
91
    
92
    for _ in 0..e {
93
        n <<= 1;
94
        if (n & 0x100000000) != 0 {
95
            n ^= polynomial;  // polynomial already has bit 32 set
96
        }
97
    }
98
    
99
    if reflected {
100
        bit_reverse_16(n) >> 31  // Reverse 16 bits, align
101
    } else {
102
        n << 32  // Shift to upper 32 bits
103
    }
104
}
105
106
/// Computes Barrett reduction constant (μ) for CRC-16
107
fn crc16_mu(polynomial: u64, reflected: bool) -> u64 {
108
    // Same algorithm as CRC-32 mu
109
    let mut n: u64 = 0x100000000;
110
    let mut q: u64 = 0;
111
    
112
    for _ in 0..33 {
113
        q <<= 1;
114
        if n & 0x100000000 != 0 {
115
            q |= 1;
116
            n ^= polynomial;
117
        }
118
        n <<= 1;
119
    }
120
    
121
    if reflected {
122
        bit_reverse(q) >> 31
123
    } else {
124
        q
125
    }
126
}
127
128
/// Formats CRC-16 polynomial for PCLMULQDQ operations
129
fn crc16_polynomial(polynomial: u64, reflected: bool) -> u64 {
130
    if !reflected {
131
        // Forward: polynomial << 16 with bit 32 set
132
        (polynomial << 16) | (1u64 << 32)
133
    } else {
134
        // Reflected: bit-reverse 16-bit poly, shift left 1, set LSB
135
        let reversed = bit_reverse_16(polynomial as u16);
136
        ((reversed as u64) << 1) | 1
137
    }
138
}
139
```
140
141
### 2. Width16 Implementation (`src/structs.rs`)
142
143
Add a new width type for CRC-16:
144
145
```rust
146
/// CRC-16 width implementation
147
#[derive(Clone, Copy)]
148
pub struct Width16;
149
150
impl CrcWidth for Width16 {
151
    const WIDTH: u32 = 16;
152
    type Value = u16;
153
}
154
```
155
156
### 3. EnhancedCrcWidth for Width16 (`src/crc16/algorithm.rs`)
157
158
The CRC-16 implementation leverages the CRC-32 infrastructure with scaling:
159
160
```rust
161
impl EnhancedCrcWidth for Width16 {
162
    fn load_constants(reflected: bool) -> [[u64; 2]; 4] {
163
        // Use CRC-32 constants - same SIMD operations
164
        crc32::consts::SIMD_CONSTANTS
165
    }
166
167
    unsafe fn create_state<T: ArchOps>(
168
        value: Self::Value,
169
        reflected: bool,
170
        ops: &T,
171
    ) -> CrcState<T::Vector> {
172
        if reflected {
173
            // Reflected: value in low position
174
            CrcState {
175
                value: ops.create_vector_from_u32(value as u32, false),
176
                reflected,
177
            }
178
        } else {
179
            // Forward: scale to 32 bits, position in high bytes
180
            let scaled = (value as u32) << 16;
181
            CrcState {
182
                value: ops.create_vector_from_u32(scaled, true),
183
                reflected,
184
            }
185
        }
186
    }
187
188
    unsafe fn extract_result<T: ArchOps>(
189
        vector: T::Vector,
190
        reflected: bool,
191
        ops: &T,
192
    ) -> Self::Value {
193
        let [low, _high] = ops.extract_u64s(vector);
194
        if reflected {
195
            // Reflected: extract from appropriate position
196
            ((low >> 32) & 0xFFFF) as u16
197
        } else {
198
            // Forward: scale back from 32 bits
199
            ((low >> 48) & 0xFFFF) as u16
200
        }
201
    }
202
203
    // fold_16, fold_width, barrett_reduction delegate to CRC-32 implementations
204
    // with appropriate value scaling
205
}
206
```
207
208
### 4. Architecture Dispatcher Update (`src/arch/mod.rs`)
209
210
Update the dispatcher to handle width=16:
211
212
```rust
213
pub(crate) unsafe fn update(state: u64, bytes: &[u8], params: &CrcParams) -> u64 {
214
    match params.width {
215
        16 => algorithm::update::<_, Width16>(state as u16, bytes, params, ops) as u64,
216
        32 => algorithm::update::<_, Width32>(state as u32, bytes, params, ops) as u64,
217
        64 => algorithm::update::<_, Width64>(state, bytes, params, ops),
218
        _ => panic!("Unsupported CRC width: {}", params.width),
219
    }
220
}
221
```
222
223
### 5. CRC-16 Constants (`src/crc16/consts.rs`)
224
225
Update the existing stub with complete constants:
226
227
```rust
228
pub const CRC16_IBM_SDLC: CrcParams = CrcParams {
229
    name: NAME_CRC16_IBM_SDLC,
230
    algorithm: CrcAlgorithm::Crc16IbmSdlc,
231
    width: 16,
232
    poly: 0x1021,
233
    init: 0xFFFF,
234
    refin: true,
235
    refout: true,
236
    xorout: 0xFFFF,
237
    check: 0x906E,
238
    keys: CrcKeysStorage::from_keys_fold_256(KEYS_1021_REVERSE),
239
};
240
241
pub const CRC16_T10_DIF: CrcParams = CrcParams {
242
    name: NAME_CRC16_T10_DIF,
243
    algorithm: CrcAlgorithm::Crc16T10Dif,
244
    width: 16,
245
    poly: 0x8BB7,
246
    init: 0x0000,
247
    refin: false,
248
    refout: false,
249
    xorout: 0x0000,
250
    check: 0xD0DB,
251
    keys: CrcKeysStorage::from_keys_fold_256(KEYS_8BB7_FORWARD),
252
};
253
```
254
255
## Data Models
256
257
### CRC-16 Polynomial Representation
258
259
| Variant | Polynomial | Scaled Polynomial (32-bit) | Reflected Polynomial |
260
|---------|------------|---------------------------|---------------------|
261
| T10-DIF | 0x8BB7 | 0x18BB70000 | N/A (forward) |
262
| IBM-SDLC | 0x1021 | 0x110210000 | 0x10811 |
263
264
### Key Array Structure
265
266
The 23-key array structure remains unchanged:
267
268
| Index | Purpose | CRC-16 Value |
269
|-------|---------|--------------|
270
| 0 | Unused | 0 |
271
| 1-2 | 16-byte fold (32*3, 32*5) | Computed |
272
| 3-4 | 128-byte fold (32*31, 32*33) | Computed |
273
| 5-6 | Short fold (32*3, 32*2) | Computed |
274
| 7 | μ (Barrett constant) | Computed |
275
| 8 | Polynomial | Scaled poly |
276
| 9-20 | Progressive fold distances | Computed |
277
| 21-22 | 256-byte fold (32*63, 32*65) | Computed |
278
279
280
281
## Correctness Properties
282
283
*A property is a characteristic or behavior that should hold true across all valid executions of a system—essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
284
285
### Property 1: Forward CRC-16 Polynomial Formatting
286
287
*For any* 16-bit polynomial value P, when generating the polynomial key for forward (non-reflected) CRC-16, the result SHALL equal `(P << 16) | (1 << 32)`.
288
289
**Validates: Requirements 1.5**
290
291
### Property 2: Reflected CRC-16 Polynomial Formatting
292
293
*For any* 16-bit polynomial value P, when generating the polynomial key for reflected CRC-16, the result SHALL equal `(bit_reverse_16(P) << 1) | 1`.
294
295
**Validates: Requirements 1.6**
296
297
### Property 3: CRC-16 Computation Matches Reference
298
299
*For any* valid CRC-16 parameters (polynomial, init, refin, refout, xorout) and any input byte sequence, the computed CRC-16 checksum SHALL match the result from the `crc` crate reference implementation.
300
301
**Validates: Requirements 2.1, 2.2, 5.5, 6.1, 6.2, 6.3**
302
303
### Property 4: CRC-32 and CRC-64 Backwards Compatibility
304
305
*For any* existing CRC-32 or CRC-64 configuration and any input byte sequence, the computed checksum SHALL match the result from the `crc` crate reference implementation, ensuring no regression from CRC-16 changes.
306
307
**Validates: Requirements 3.1, 3.2, 3.4**
308
309
### Property 5: CRC-16 Checksum Combination Round-Trip
310
311
*For any* valid CRC-16 parameters and any two input byte sequences A and B, `checksum_combine(checksum(A), checksum(B), len(B))` SHALL equal `checksum(A + B)`.
312
313
**Validates: Requirements 6.4**
314
315
## Error Handling
316
317
### Invalid Width
318
319
When `CrcParams::new()` is called with an unsupported width (not 16, 32, or 64), the function SHALL panic with a descriptive error message.
320
321
### Invalid Polynomial
322
323
CRC-16 polynomials must fit in 16 bits. If a polynomial larger than 0xFFFF is provided for width=16, the behavior is undefined (caller's responsibility to provide valid parameters).
324
325
### Empty Input
326
327
Empty input (`&[]`) is valid and SHALL return the initial CRC value XORed with xorout, consistent with existing CRC-32 and CRC-64 behavior.
328
329
## Testing Strategy
330
331
### Unit Tests
332
333
Unit tests will verify specific examples and edge cases:
334
335
1. **Check Value Tests**: Verify "123456789" produces correct check values for CRC-16/IBM-SDLC (0x906E) and CRC-16/T10-DIF (0xD0DB)
336
2. **Key Generation Tests**: Verify generated keys match reference values in `KEYS_8BB7_FORWARD` and `KEYS_1021_REVERSE`
337
3. **Empty Input Tests**: Verify empty input returns expected value
338
4. **Constant Definition Tests**: Verify CRC16_IBM_SDLC and CRC16_T10_DIF have correct parameter values
339
340
### Property-Based Tests
341
342
Property-based tests will use the `proptest` crate (already used in the project) with minimum 100 iterations per property:
343
344
1. **Property 1 & 2**: Generate random 16-bit polynomials and verify polynomial key formatting
345
2. **Property 3**: Generate random CRC-16 parameters and random byte sequences, compare against `crc` crate
346
3. **Property 4**: Use existing CRC-32/64 configurations with random byte sequences, compare against `crc` crate
347
4. **Property 5**: Generate random CRC-16 parameters and two random byte sequences, verify combination
348
349
### Test Configuration
350
351
```rust
352
// Property test configuration
353
proptest! {
354
    #![proptest_config(ProptestConfig::with_cases(100))]
355
    
356
    // Feature: crc16-hardware-acceleration, Property 1: Forward polynomial formatting
357
    #[test]
358
    fn prop_forward_polynomial_formatting(poly in 0u16..=0xFFFFu16) {
359
        let result = crc16_polynomial(poly as u64, false);
360
        let expected = ((poly as u64) << 16) | (1u64 << 32);
361
        prop_assert_eq!(result, expected);
362
    }
363
}
364
```
365
366
### Regression Testing
367
368
All existing tests in `src/arch/mod.rs`, `src/test/`, and `tests/` must continue to pass after CRC-16 implementation.