Add GGUF example to low level API example #973

examples/78_Accessing_Low_Level_Vector_APIs.ipynb

@@ -32,7 +32,7 @@
    "outputs": [],
    "source": [
     "%%capture\n",
-    "!pip install git+https://github.com/neuml/txtai#egg=txtai[ann]"
+    "!pip install git+https://github.com/neuml/txtai#egg=txtai[ann] gguf"
    ]
   },
   {
@@ -343,6 +343,112 @@
     "Same result. Note the score is slightly different but this is expected."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "89abb301",
+   "metadata": {},
+   "source": [
+    "# GGUF Support\n",
+    "\n",
+    "`txtai` 9.1 also adds support for [GGML](https://github.com/ggml-org/ggml) / [GGUF](https://huggingface.co/docs/hub/en/gguf) popularized by [llama.cpp](https://github.com/ggml-org/llama.cpp)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "e04498fc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ann = ANNFactory.create({\n",
+    "    \"backend\": \"ggml\",\n",
+    "    \"ggml\": {\n",
+    "        \"quantize\": \"Q4_0\"\n",
+    "    }\n",
+    "})\n",
+    "ann.index(data)\n",
+    "ann.save(\"vectors.gguf\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7bea0f58",
+   "metadata": {},
+   "source": [
+    "Now let's check out the generated file using the [gguf](https://github.com/ggml-org/llama.cpp/tree/master/gguf-py) package provided by llama.cpp."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "b0e6f85e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tensor Name                    | Shape           | Size         | Quantization\n",
+      "--------------------------------------------------------------------------------\n",
+      "data                           | 384x1411868     | 258.52 MB    | Q4_0\n"
+     ]
+    }
+   ],
+   "source": [
+    "from gguf.gguf_reader import GGUFReader\n",
+    "\n",
+    "reader = GGUFReader(\"vectors.gguf\")\n",
+    "\n",
+    "# List all tensors\n",
+    "info = \"{:<30} | {:<15} | {:<12} | {}\"\n",
+    "print(info.format(\"Tensor Name\", \"Shape\", \"Size\", \"Quantization\"))\n",
+    "print(\"-\" * 80)\n",
+    "for tensor in reader.tensors:\n",
+    "    shape = \"x\".join(map(str, tensor.shape))\n",
+    "    size = f\"{tensor.n_elements / 2 / 1024 / 1024:.2f} MB\"\n",
+    "    quantization = tensor.tensor_type.name\n",
+    "    print(info.format(tensor.name, shape, size, quantization))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7be79bf7",
+   "metadata": {},
+   "source": [
+    "And search like we did with Torch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "450dc024",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The answer to your question, that how many miles is it from earth to mars, is very easy to know. Because of huge satellites which are being sent to\n",
+      "mars in search of life from many countries, we have discovered a lot about mars. According to experts, earth and mars reaches to their closest points\n",
+      "in every 26 months. This situation is considered as opposition of mars as the location of sun and mars in totally opposite to each other in relation\n",
+      "to earth. When this opposition takes place, the planet is visible with a red tint in the sky from earth. And this also gives mars a name, i.e. the red\n",
+      "planet. Mars is also the fourth planet from sun, which is located between Jupiter and earth. Its distance from sun is not only opposite but is also\n",
+      "much further away, than that of the earth and sun. The distance between the sun and mars is said to be 140 million miles. Mars can reach about 128\n",
+      "million miles closer to the sun whereas it can even travel around 154 million miles away from it. The assumed distance between mars and earth is said\n",
+      "to be between 40 to 225 million miles. The distance between these two planets keeps on changing throughout the year because of the elliptical path in\n",
+      "which all the planets rotate. As the distance between mars, sun and earth is so much high, it takes a Martian year, for mars to go around the sun. The\n",
+      "Martian period includes a time of around 687 earth days. This means that, it takes more than 2 years for the mars to reach its initial rotation point.\n",
+      "If we talk about one Martian day, it is the total time which is taken by a planet to spin around once. This day usually lasts longer than our regular\n",
+      "earth days. So this was the actual reason which states the distance between earth and mars. \n",
+      "\n",
+      " 0.7043964862823486\n"
+     ]
+    }
+   ],
+   "source": [
+    "search(\"How far is earth from mars?\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "af1a344c",
@@ -355,8 +461,22 @@
   }
  ],
  "metadata": {
+  "kernelspec": {
+   "display_name": "local",
+   "language": "python",
+   "name": "python3"
+  },
   "language_info": {
-   "name": "python"
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.19"
   }
  },
  "nbformat": 4,