Technical Details
Operations:
29.4 GOPs per input token
Context Length:
2048
Numerical Scheme:
A8W4, symmetric, group-wise
Inference API:
C++, Python, Hailo-Ollama
Compiled Model:
Performance Metrics
Load Time In Sec
4.31
Time To First Token In Sec
0.35
TPS
9.79