txtai/examples/74_OpenAI_Compatible_API.ipynb

OpenAI Compatible API

txtai has long been able host a FastAPI based service. There are clients for Python, JavaScript, Java, Rust, Go.

The API service also supports hosting OpenAI-compatible API endpoints. A standard OpenAI client can then be used to connect to a txtai service. This enables quickly trying txtai with a familiar-to-use client. It's also a way to do local/offline development testing using the OpenAI client.

This notebook will walk through comprehensive examples.

Start API service

For this notebook, we'll run txtai through Docker.

Save the following to /tmp/config/config.yml.

config.yml

# Enable OpenAI compat endpoint
openai: True

# Load Wikipedia Embeddings index
cloud:
  provider: huggingface-hub
  container: neuml/txtai-wikipedia

# LLM instance
llm:
  path: llava-hf/llava-interleave-qwen-0.5b-hf

# RAG pipeline configuration
rag:
  path: Qwen/Qwen3-4B-Instruct-2507
  output: flatten
  system: You are a friendly assistant. You answer questions from users.
  template: |
    Answer the following question using only the context below. Only include information
    specifically discussed.

    question: {question}
    context: {context}

# Text to Speech
texttospeech:
  path: neuml/kokoro-fp16-onnx

# Transcription
transcription:
  path: distil-whisper/distil-large-v3

Start Docker service.

docker run -it -p 8000:8000 -v /tmp/config:/config -e CONFIG=/config/config.yml \
--entrypoint uvicorn neuml/txtai-gpu --host 0.0.0.0 txtai.api:app

Alternatively, txtai can be directly installed and run as follows:

pip install txtai[all]
CONFIG=/tmp/config/config.yml uvicorn "txtai.api:app"

The API has token-based authorization built-in. Read more on that here.

Run a text chat completion

The first example will run a text chat completion. The model is a RAG pipeline - this is more sophisticated than just a simple LLM call!

Agents, pipelines and workflows can all be run through this interface!

In [1]:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="api-key",
)

response = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": "Tell me about the iPhone",
    }],
    model="rag",
    stream=True
)

for chunk in response:
    if chunk.choices:
        print(chunk.choices[0].delta.content, end="")

The iPhone is a line of smartphones designed and marketed by Apple Inc. that uses Apple's iOS mobile operating system. The first-generation iPhone was announced by former Apple CEO Steve Jobs on January 9, 2007. 

Since then, Apple has annually released new iPhone models and iOS updates. The most recent models being the iPhone 16 and 16 Plus, and the higher-end iPhone 16 Pro and 16 Pro Max. 

More than 2.3 billion iPhones have been sold as of January 1, 2024, making Apple the largest vendor of mobile phones in 2023.

As mentioned above, the model supports much of what's available in txtai. For example, let's run a chat completion that runs an embeddings search.

In [2]:

response = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": "Tell me about the iPhone",
    }],
    model="embeddings",
)

print(response.choices[0].message.content)

The iPhone is a line of smartphones designed and marketed by Apple Inc. that uses Apple's iOS mobile operating system. The first-generation iPhone was announced by former Apple CEO Steve Jobs on January 9, 2007. Since then, Apple has annually released new iPhone models and iOS updates. iPhone naming has followed various patterns throughout its history.

Vision models

Any supported txtai LLM can be run through the chat completion API. Let's run an example that describes an image.

In [3]:

response = client.chat.completions.create(
    model="llm",
    messages = [{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this image?"},
            {"type": "image_url", "image_url": {
                "url": "https://raw.githubusercontent.com/neuml/txtai/master/logo.png",
            }}
        ]}
    ]
)
print(response.choices[0].message.content)

The image shows a logo with the text "Txtai" in blue and green colors. The

Embeddings API

Next let's generate embeddings.

In [4]:

for x in client.embeddings.create(input="This is a test", model="vectors").data:
    print(x)

Embedding(embedding=[-0.01969079300761223, 0.024085085839033127, 0.0043829963542521, -0.027423616498708725, 0.040405914187431335, 0.017446696758270264, 0.028464825823903084, 0.000792442646343261, -0.03107883222401142, -0.024745089933276176, -0.013542148284614086, 0.039981111884117126, -0.01401221938431263, -0.011294773779809475, -0.04346214607357979, 0.015698621049523354, 0.03775031119585037, -0.009020405821502209, 0.046784739941358566, -0.017400527372956276, -0.0670166090130806, -0.05122058466076851, 0.027725063264369965, -0.023947732523083687, -0.044582683593034744, 0.04960233345627785, 0.029517438262701035, 0.05424104258418083, -0.06027599796652794, -0.035852570086717606, 0.01336587406694889, -0.008941668085753918, 0.00014064145216252655, -0.05230511724948883, -0.02150369994342327, 0.04969678074121475, -0.05967864394187927, -0.029450856149196625, -0.01113089732825756, -0.01256561279296875, -0.012282170355319977, 0.03466389700770378, -0.005313237197697163, -0.037443146109580994, -0.04366842657327652, -0.019057273864746094, -0.04015717655420303, 0.050483088940382004, -0.011932676658034325, -0.026569517329335213, -0.048395730555057526, 0.021978085860610008, 0.03273716941475868, -0.009176520630717278, -0.05367470160126686, 0.01982428878545761, -0.00373812741599977, 0.009933742694556713, 0.044389136135578156, -0.06162404641509056, 0.03372818976640701, 0.006638737395405769, 0.029836857691407204, 0.014663859270513058, 0.04531872272491455, 0.03151382878422737, -0.007935297675430775, -0.02053055912256241, -0.06477595120668411, -0.017908524721860886, -0.014721200801432133, -0.0072138686664402485, -0.03244556114077568, -0.018965184688568115, 0.04862097278237343, -0.02961636148393154, 0.005204972345381975, 0.015699708834290504, 0.05033862590789795, -0.017976371571421623, -0.05143386870622635, -0.014295309782028198, -0.018152274191379547, 0.04641849547624588, 0.007279090117663145, -0.0060980431735515594, -0.04208022356033325, 0.05402654781937599, 0.0001357585861114785, 0.044958386570215225, -0.03261513262987137, 0.02126067876815796, 0.020893605425953865, -0.007570710498839617, -0.015284491702914238, -0.011333705857396126, -0.006006874144077301, 0.03481211140751839, 0.04163122922182083, -0.0683935135602951, -0.030256368219852448, -0.024272358044981956, 0.04630593582987785, -0.05253031477332115, 0.011599390767514706, -0.034757863730192184, 0.0033751465380191803, -0.03200560435652733, -0.04386962205171585, 0.015501669608056545, -0.01703309454023838, -0.029905665665864944, -0.03208091855049133, -0.027883553877472878, 0.007325653452426195, 0.03735042363405228, 0.08069189637899399, -0.044986918568611145, 0.030896944925189018, 0.017477652058005333, -0.0063758366741240025, 0.02287706732749939, 0.016398170962929726, 0.01946086250245571, 0.012854589149355888, 0.04439576715230942, 0.04581235349178314, -0.0008034319616854191, -0.03105296567082405, -0.024504775181412697, 0.023659462109208107, 0.04492054134607315, -0.025883017107844353, -0.002515115775167942, 0.052770763635635376, 0.009667950682342052, -0.022283289581537247, -0.07817766815423965, 0.03883073106408119, -0.04804662615060806, 0.011968107894062996, -0.03163604810833931, 0.030380938202142715, -0.022775596007704735, 0.03142687678337097, -0.11540865898132324, -0.062065351754426956, 0.003252241527661681, -0.016604064032435417, 0.046795569360256195, -0.01973356492817402, 0.005612187087535858, 0.04902602732181549, -0.029760321602225304, -0.0006560107576660812, 0.02137850970029831, 0.021465344354510307, -0.030499190092086792, -0.013952907174825668, 0.015388991683721542, -0.004734670277684927, -0.02678225375711918, 0.056917935609817505, -0.0031489196699112654, -0.000562859873753041, 0.08021821081638336, 0.045039497315883636, 0.051955677568912506, -0.06851264089345932, -0.0202163215726614, -0.020257024094462395, 0.009915929287672043, 0.027132542803883553, -0.039319392293691635, -0.06750114262104034, 0.00721193291246891, 0.011379252187907696, -0.00012379158579278737, 0.021098755300045013, -0.017165066674351692, -0.06655416637659073, 0.03575438633561134, -0.0449126660823822, 0.024580610916018486, 0.0027450474444776773, -0.07029049843549728, -0.0058233728632330894, -0.0031324869487434626, -0.022562572732567787, -0.002092051785439253, -0.01972377672791481, -0.014447340741753578, 0.02001781575381756, -0.04224644973874092, 0.08794320374727249, -0.05012425035238266, -0.03000028431415558, -0.006967171560972929, -0.0206689964979887, 0.042854372411966324, 0.018307263031601906, 0.04896565154194832, 0.025682201609015465, -0.013927857391536236, -0.026135331019759178, 0.05985535308718681, -0.022972915321588516, -0.06837267428636551, 0.03858938440680504, 0.01297465804964304, -0.01869095303118229, -0.014788917265832424, 0.05812034010887146, -0.005296449176967144, -0.03188127279281616, -0.014335273765027523, 0.029694614931941032, -0.006149643566459417, 0.0199541375041008, -0.04401557520031929, 0.08680693805217743, 0.02373044192790985, -0.05719068646430969, 0.00026498522493056953, -0.047968123108148575, 0.05128588527441025, 0.08984201401472092, 0.018948959186673164, -0.019343748688697815, -0.02114059403538704, -0.000319077109452337, -0.0483400821685791, 0.02235756441950798, -0.04526951164007187, -0.016685402020812035, 0.04920167103409767, 0.0009292830363847315, 0.0066963727585971355, 0.06434790045022964, -0.07675006985664368, 0.025055741891264915, 0.039694759994745255, -0.04413995519280434, 0.053703855723142624, 0.022806784138083458, -0.02683648094534874, 0.04088520258665085, -0.02505207061767578, 0.038970883935689926, -0.011933756060898304, 0.017762111499905586, -0.052576545625925064, -0.02732933685183525, 0.024120833724737167, -0.011316879652440548, -0.04519795626401901, 0.012005027383565903, 0.016074027866125107, -0.019522851333022118, 0.07912492007017136, -0.010790158063173294, 0.003584112972021103, -0.018683504313230515, -0.03872854635119438, -0.0293426550924778, -0.028616394847631454, 0.0034447587095201015, 0.008824280463159084, 0.0267381239682436, -0.014405295252799988, 0.01340708788484335, 0.022090492770075798, 0.041456740349531174, 0.01306570041924715, 0.012696513906121254, -0.05636722221970558, 0.05526677146553993, 0.014159836806356907, -0.05075988918542862, -0.03631533309817314, 0.04115152731537819, 0.06140957027673721, -0.06539256870746613, -0.01610933430492878, 0.08129005879163742, -0.054096464067697525, 0.021539339795708656, -0.009134260006248951, 0.04177645593881607, 0.026524635031819344, 0.016892578452825546, 0.037963252514600754, -0.06906059384346008, 0.050708942115306854, 0.06792867928743362, -0.0004703162703663111, 0.018694648519158363, -0.031178174540400505, -0.03567223250865936, -0.035771071910858154, 0.05392008647322655, 0.06253240257501602, -0.020289720967411995, 0.034436099231243134, 0.03414503112435341, 0.0034774248488247395, -0.04452746734023094, -0.03509671986103058, -0.10872040688991547, 0.016063231974840164, 0.047865595668554306, -0.04542273283004761, 0.014507413841784, 0.0009427995537407696, -0.0031647789292037487, -0.0013884446816518903, -0.045522164553403854, 0.031990133225917816, -0.07940599322319031, -0.0021216440945863724, -0.003062204457819462, 0.0284376610070467, 0.038331907242536545, -0.021678920835256577, 0.010201317258179188, -0.01604599319398403, 0.06507452577352524, 0.0687805563211441, 0.05626540631055832, -0.035019401460886, 0.013606756925582886, 0.01355750672519207, -0.0009656146867200732, 0.008775751106441021, -0.023357177153229713, -0.027274709194898605, -0.030927035957574844, -0.014168186113238335, -0.0025208715815097094, -0.06382670253515244, 0.0016783965984359384, 0.03997219353914261, -0.011281637474894524, -0.0564236082136631, 0.0001946773991221562, -0.044997114688158035, 0.006665860302746296, -0.02552937902510166, -0.0387411043047905, -0.007421125657856464, -0.018388714641332626, 0.04417712241411209, -0.03386503830552101, -0.015952911227941513, 0.0044018859043717384, -0.03185226395726204, 0.03936305642127991, -0.0018688770942389965, -0.04392078518867493, 0.02990303561091423, -0.0194404199719429, 0.05901814624667168, -0.021767310798168182, 0.032181400805711746, 0.015370846726000309, 0.031207047402858734, -0.016042204573750496, -0.016823984682559967, -0.005706059746444225, -0.03331942856311798, 0.011479238979518414, -0.043793581426143646, 0.032494351267814636, -0.06793207675218582, 0.05236655846238136, -0.031655143946409225, 0.01929832063615322, -0.0250355564057827, -0.03658934682607651, 0.04857027530670166, -0.06623365730047226, -0.04268127307295799, -0.04363507777452469, 0.044615332037210464, -0.00559930969029665, -0.03717941418290138, 0.028203044086694717, 0.00480041466653347, 0.009005775675177574, -0.01836307905614376, 0.054084815084934235, -0.017307721078395844, 0.048483166843652725, 0.023009151220321655, -7.859049219405279e-05, 0.030783794820308685, 0.043127138167619705, 0.005765000823885202, 0.008811963722109795, -0.05386245995759964, 0.004587933421134949, -0.005802399478852749, 0.0050554038025438786, 0.03453978896141052, -0.012859856709837914, -0.01060124859213829, -0.013389998115599155, -0.04355772212147713, 0.016539031639695168, -0.05041985213756561, -0.0248723067343235, 0.08495471626520157, 0.055736441165208817, -0.019743982702493668, -0.04003654792904854, 0.05553850531578064, 0.009581065736711025, -0.020963093265891075, 0.03220677375793457, -0.012795533053576946, 0.052986159920692444, -0.05288834869861603, 0.053567126393318176, 0.01575312204658985, 0.05197490379214287, -0.012308254837989807, -0.004616653546690941, 0.005736787803471088, -0.010011504404246807, 0.010513711720705032, -0.054142292588949203, -0.06452780216932297, -0.06130351126194, 0.002477638190612197, -0.022184111177921295, -0.000995964859612286, -0.05435270443558693, 0.0074448655359447, -0.023539019748568535, -0.031608957797288895, 0.0064430260099470615, -0.030367573723196983, 0.015771696344017982, -0.014180796220898628, -0.04425235465168953, 0.06702947616577148, -0.000456854235380888, 0.010592995211482048, -0.026347137987613678, -0.03434554859995842, 0.01162006612867117, -0.00362666929140687, 0.03504545986652374, 0.002880056854337454, -0.008586738258600235, -0.0005600558361038566, -0.01934652030467987, 0.05669917166233063, -0.00034789182245731354, -0.01825639232993126, -0.012466290034353733, -0.03704797849059105, -0.002550555858761072, -0.022397562861442566, 0.020881881937384605, -0.013832776807248592, 0.027578793466091156, 0.045279063284397125, -0.000525087583810091, -0.047328196465969086, -0.007053776178508997, -0.0021893021184951067, 0.0286997202783823, 0.02384152263402939, 0.006024117581546307, 0.013520568609237671, 0.026852741837501526, 0.04367787763476372, -0.02344651333987713, -0.041360042989254, -0.027980612590909004, -0.014400728978216648, -0.0577680841088295, 0.05705561116337776, -0.00984896719455719, 0.010015214793384075, 0.0763126090168953, 0.07034917175769806, 0.011689933016896248, -0.04705473780632019, -0.028127267956733704, -0.028715649619698524, 0.00838626641780138, -0.09287010133266449, -0.05999135598540306, -0.03459229692816734, -0.03452807664871216, 0.029350021854043007, 9.120464028455899e-07, 0.006535083521157503, 0.029187319800257683, -0.06986693292856216, -0.02206997573375702, -0.05103607848286629, -0.024477669969201088, 0.020876919850707054, 0.045642390847206116, 0.04098346829414368, -0.01810697466135025, -0.018912270665168762, -0.013277142308652401, 0.0213322751224041, -0.026938313618302345, -0.05354780703783035, -0.016160599887371063, 0.0029611149802803993, -0.02684030868113041, 0.04515037313103676, 0.02446618117392063, -0.02725314162671566, -0.024469705298542976, 0.021647747606039047, 0.002507369965314865, -0.04194789007306099, 0.017087087035179138, 0.0518130287528038, 0.05085260793566704, -0.07700842618942261, 0.0056351562961936, 0.060032691806554794, 0.006674149073660374, -0.05446042865514755, -0.04615267738699913, 0.024369537830352783, 0.0271424762904644, -0.012347695417702198, 0.060294460505247116, -0.016132934018969536, 0.017447318881750107, 0.05670442432165146, 0.0015670768916606903, 0.0686553418636322, -0.026241859421133995, -0.015325505286455154, -0.01770787686109543, 0.02104933187365532, -0.046040672808885574, 0.025931548327207565, 0.038434647023677826, 0.026901748031377792, 0.009451834484934807, -0.036178428679704666, -0.017309701070189476, 0.025584906339645386, 0.061249326914548874, -0.09514082968235016, 0.00902795884758234, 0.04994441568851471, -0.0038626997265964746, -0.011658617295324802, 0.008113766089081764, -0.014046317897737026, -0.011056281626224518, 0.04113991931080818, -0.033263616263866425, -0.06105482578277588, -0.031283922493457794, -0.007570411544293165, -0.0032288332004100084, 0.0004964112886227667, -0.002208895515650511, 0.024621492251753807, 0.02076159231364727, 0.009394104592502117, -0.05932564660906792, 0.020849449560046196, 0.03622383624315262, -0.05445172265172005, -0.01577809825539589, -0.016015635803341866, -0.034171875566244125, 0.07400329411029816, 0.06425172090530396, 0.013667335733771324, -0.027485249564051628, -0.06779397279024124, 0.011731340549886227, -0.021519936621189117, 0.05336484685540199, 0.09234699606895447, -0.025581158697605133, 0.04105791822075844, 0.0033194669522345066, 0.009193984791636467, 0.01257328037172556, 0.009872003458440304, 0.013460072688758373, -0.011782833375036716, 0.056569892913103104, 0.011185556650161743, -0.001791957183741033, 0.02985113114118576, 0.03551100194454193, 0.0525372251868248, 0.009313385002315044, 0.029556646943092346, 0.010092461481690407, -0.04109922796487808, -0.09827771782875061, 0.07528837770223618, 0.018835244700312614, -0.02083747275173664, -0.04701956734061241, -0.0014823883539065719, -0.003127161879092455, -0.03790943697094917, -0.05166167765855789, 0.015131598338484764, -0.005340536590665579, -0.014197085052728653, 0.05665569752454758, 0.006526419427245855, -0.02720179408788681, 0.00903793890029192, 0.07492761313915253, -0.04322653263807297, -0.05390876159071922, 0.03704892471432686, -0.027291180565953255, -0.0566035658121109, 0.015606636181473732, 0.027065519243478775, -0.0017480048118159175, 0.027853772044181824, -0.006371915340423584, 0.030529864132404327, 0.018552660942077637, 0.034608323127031326, 0.018036214634776115, 0.03474709764122963, 0.010607247240841389, 0.008939945138990879, 0.005929744802415371, -0.025183523073792458, -0.0025635838974267244, -0.0645676925778389, 0.0062942770309746265, 0.043695416301488876, 0.011311094276607037, 0.006157045252621174, -0.0021617324091494083, 0.03866882994771004, -0.058823224157094955, 0.06246255710721016, 0.0071550337597727776, 0.0022470480762422085, -0.008880370296537876, 0.03494860604405403, 0.038959626108407974, 0.04550785943865776, 0.030317384749650955, -0.00612551299855113, 0.08027740567922592, 0.0028502782806754112, -0.008108875714242458, -0.029123658314347267, -0.012007949873805046, -0.014279266819357872, -0.02980010211467743, 0.02040782757103443, 0.06390708684921265, -0.0006256934138946235, -0.03723321482539177, -0.013057096861302853, 0.04114076867699623, -0.017182866111397743, -0.05549640208482742, 0.02064032293856144, 0.01683172956109047, -0.008635859936475754, 0.03218064829707146, 0.04564550518989563, -0.021377939730882645, 0.021940747275948524, 0.020410453900694847, 0.017982320860028267, 0.02150171808898449, 0.05921953544020653, -0.042486630380153656, -0.017924992367625237, -0.0114266537129879, -0.02765769325196743, 0.02116318792104721, -0.0008785029058344662, 0.00839359499514103, 0.007519723381847143, -0.07929962873458862, 0.01306573860347271, 0.00335461413487792, -0.013990496285259724, 0.00019492211868055165, -0.017358528450131416, -0.03889889642596245, -0.008545472286641598, 0.01378809567540884, 0.06300467997789383, 0.05205303058028221, 0.029774265363812447, 0.05180739611387253, -0.04484200477600098, -0.03888325020670891, -0.056330904364585876, 0.004683728329837322, 0.016883134841918945, -0.03816996142268181, 0.01605170965194702, 0.0022271168418228626, 0.0010828975355252624, 0.038834843784570694, 0.019416887313127518, 0.00031489337561652064, 0.05024728924036026, -0.05813521891832352, -0.006695288233458996, 0.042213670909404755, -0.012247920036315918, 0.028528228402137756, -0.02632697857916355, -0.05482589080929756, 0.00981950294226408, 0.02605678141117096, 0.06638345867395401, -0.018992368131875992, 0.04858163744211197, -0.014409814961254597, -0.0310173612087965, -0.05839765444397926, 0.08313969522714615, 0.05511852726340294, 0.047723494470119476, -0.033163223415613174, -0.040427759289741516, 0.011779758147895336, -0.05743969976902008, -0.021088508889079094, -0.018184570595622063, 0.022849485278129578, -0.010282794013619423, -0.010582848452031612, -0.038172293454408646, -0.02383989654481411, -0.047329485416412354, -0.02541566826403141, 0.027357304468750954, -0.06858660280704498, -0.06362185627222061, -0.0027012284845113754, -0.035492997616529465, -0.06344638019800186, 0.03718043491244316, 0.012817914597690105, 0.018238751217722893, -0.007895039394497871, 0.042976900935173035, -0.06253521889448166, 0.02173938974738121, 0.01422695629298687, 0.06118226796388626], index=0, object='embedding')

This uses the vector model associated with the current embeddings database.

Text to Speech (TTS)

This API can do more than just work with text. Let's generate speech.

In [5]:

from IPython.display import Audio, display

with client.audio.speech.with_streaming_response.create(
    model="neuml/kokoro-fp16-onnx",
    input="txtai is an all-in-one embeddings database for semantic search, LLM orchestration and semantic workflows",
    voice="bm_lewis",
) as response:
    response.stream_to_file(file="out.mp3")

display(Audio("out.mp3"))

Your browser does not support the audio element.

Transcription

The generated speech can also be transcribed back to text.

In [6]:

f = open("out.mp3", "rb")
client.audio.transcriptions.create(
    model="whisper",
    file=f
).text

Out [6]:

"Text AI is an all in one embedding's database for semantic search, LLM orchestration and semantic workflows."

JavaScript client

Given that this is an OpenAI-compatible API, other OpenAI clients are also supported. Let's try a few examples with the JavaScript client.

Install via npm install openai

In [7]:

%%writefile chat.js

import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: "http://localhost:8000/v1",
    apiKey: "api-key"
});

async function main() {
    const stream = await openai.chat.completions.create({
        model: "rag",
        messages: [{ role: "user", content: "Tell me about the iPhone" }],
        stream: true,
    });

    for await (const chunk of stream) {
        process.stdout.write(chunk.choices[0]?.delta?.content || "");
    }
}

main();

Overwriting chat.js

In [8]:

!node --no-warnings chat.js

The iPhone is a line of smartphones designed and marketed by Apple Inc. that uses Apple's iOS mobile operating system. The first-generation iPhone was announced by former Apple CEO Steve Jobs on January 9, 2007. As of January 1, 2024, more than 2.3 billion iPhones have been sold, making Apple the largest vendor of mobile phones in 2023.

As we can see, this is the same output as we had earlier with the Python client.

Let's try generating speech.

In [ ]:

%%writefile speech.js

import fs from "fs";
import path from "path";
import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: "http://localhost:8000/v1",
    apiKey: "api-key"
});

const speechFile = path.resolve("./speech.mp3");

async function main() {
  const mp3 = await openai.audio.speech.create({
    model: "neuml/kokoro-fp16-onnx",
    input: "txtai is an all-in-one embeddings database for semantic search, LLM orchestration and semantic workflows",
    voice: "bm_lewis",
  });

  const buffer = Buffer.from(await mp3.arrayBuffer());
  await fs.promises.writeFile(speechFile, buffer);
}

main();

Overwriting speech.js

In [ ]:

!node --no-warnings speech.js

In [11]:

display(Audio("speech.mp3"))

Your browser does not support the audio element.

Speech is the same as above, as expected.

In [12]:

%%writefile transcribe.js

import fs from "fs";
import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: "http://localhost:8000/v1",
    apiKey: "api-key"
});

async function main() {
  const transcription = await openai.audio.transcriptions.create({
    file: fs.createReadStream("speech.mp3"),
    model: "whisper",
  });

  console.log(transcription.text);
}

main();

Overwriting transcribe.js

In [13]:

!node --no-warnings transcribe.js

Text AI is an all in one embedding's database for semantic search, LLM orchestration and semantic workflows.

Wrapping up

This notebook covered how to setup an OpenAI-compatible API endpoint for txtai. It enables quickly trying txtai with a familiar-to-use client. It's also a way to do local/offline development testing using the OpenAI client. Just another way to make it easier to use txtai!