IATESIS NOETILUM · APPLIED RESEARCH Checking index

NOETILUM · RESEARCH IN ARTIFICIAL INTELLIGENCE

Artificial intelligence
to explore
libraries and theses.

IATESIS is part of Noetilum's applied research. We process bibliographic data and train a neural encoder to investigate information retrieval with AI. Search references and thesis passages, compare methods and review the sources.

— records in index
3 ways to explore
PDF query text and pages
Start a search
IATESIS · THE IMPLEMENTED PROCESS See methodology ↗
  1. Prepare two collections

    Catalog TXT → 593,067 JSONL records → BM25 index.

    PDF Text per page → 6,541 fragments → BM25 index.

  2. Train the encoder with titles

    A selection of 12,000 records; 8,750 titles update the weights over 4 epochs. Epoch 3 is retained based on validation.

    TF-IDF: 4,096 → linear layer: 1,024 → L2 normalization

  3. Calculate vectors with fixed weights

    The same encoder represents catalog titles and PDF passages. PDF passages reuse the weights learned from titles.

  4. Retrieve from the selected collection

    Question → BM25, dense search or RRF combination → references or passages with page and source.

  5. Show sources and draft on request

    Ollama runs Llama with the question and retrieved evidence. This project did not train Llama on the theses.

Documented delivery summary: 50 PDFs reviewed, 49 with fragments and 32 with partial extraction. Functional testing does not equate to human relevance assessment.

RESEARCH PROJECT

NOETILUM

Research and Applied Intelligence

About Noetilum ↗

RESEARCH LINE

Data and neural networks

Information processing, encoder training and source retrieval.

EXPLORE WITH US

AI applied to knowledge

A demonstration to explore with clients how to consult documents and check results.

YOUR STARTING POINT

What do you want to discover?

Try without filters

Combines word matching and vector similarity using RRF.

Narrow your search Optional

Just the name of the faculty. To search for a person, use the query field.

⌘ / Ctrl + Enter to search

In “Catalogue” bibliographic records are consulted. Switch to "PDF" to search for extracted passages and open the thesis page.

Titles and passages are shown in their original language. Spanish keywords usually work best with this collection.

Your discovery space

PUBLIC CONSULTATION

EXPLORE THE CATALOG WITH AI

Start with a query.

Write a topic or the name of an author.
Here you will find references and their sources.

01Find

Search by words or similarity.

02Explore

Check titles, authors and links.

03Connect

Ask Ollama to draft an answer.

IATESIS · UNDERSTAND THE PROJECT, STEP BY STEP

Technical and scientific manual of the project.

A guide from data to answer: applied methodology, parameters, code snippets and evidence. Documented delivery of September 14, 2026.

593.067 catalog records for this delivery
12,000 selected to prepare for training
49 PDF with indexed text, from 50 files reviewed
6,541 PDF fragments available

The full catalog and the training sample have different functions. The figures on this sheet correspond to the indicated delivery; the senior accountant consults the catalog that is active.

CHAPTER 01 How to read the manual and check its claims

The laboratory question is how to transform bibliographic records and thesis text into searchable sources, and how to use those sources to support an answer. The available result is an executable and traceable system. Determining how much it improves the relevance or work of a library requires another stage: an evaluation with defined people and criteria.

To start without knowledge of AI, it is worth distinguishing five actions. Prepare converts data to a consistent structure. Index organizes that structure to find results. Train modifies parameters using examples and a loss. Vectorize applies an already set transformation and saves numbers by text. Generate produces new text from a question and context. In this project only the title encoder training adjusted a network; the other actions have different objectives.

Each section connects input → operation → parameters → output file → verification. The code blocks identified as literals come from the mentioned functions. Abbreviated examples, pedagogical formulas and commands for another installation are indicated as such: they are not records of new executions.

Status of an assertion How to interpret it in this manual
Applied and proven There is evidence in the code and artifacts or logs of this release
Implemented, not activated A code path is available; no results are attributed to it
Configurable The value can be changed by the indicated argument or variable; other values ​​require rechecking
Research proposal A hypothesis or extension is proposed; It is not executed nor is an improvement promised

Data figures were checked against artifacts/entrega-servidor-entrenada-v1/payload/BUILD_REPORT.json , manifests, SQLite databases and sectesis/neural/encoder.json . The private Release of artifacts retains the preparation code 9193ce88f586783d3d8ab5dd93ec6880f49116a9 . Function citations are pinned to the audited commit code 01bbc0c , so that future releases of main do not move its lines. Manual revision does not execute new training.

GitHub requests access to the private repository to open those sources. The excerpts and explanations on this page allow you to follow the process without that access. Local operational logs are not published as if they were repository data. An integrity test demonstrates that the bytes match; a functional test demonstrates that an operation responds; none alone is equivalent to scientific quality or human validation.

Reading path: start with counts and diagram; loop through TXT, PDF, BM25, training and vectors; then study the consultation, the evaluation and Llama. The API and server sections explain the operation. At the end, contributions, pending research and a file map are presented to continue reading the code. The glossary allows you to consult the terms during that tour.

The portal opens in English by default and also offers Spanish. A language cookie remembers your choice. The interface and this manual are served from translated local files; queries and documents are not sent to translation services. Titles, citations, passages and literal code examples remain in their original language. Spanish keywords usually work best with this collection. Ollama is instructed to draft in the selected language, although the quality and faithfulness of its answer require human review.

The search API accepts the optional language field, with values en or es, for both catalog and PDF searches. When omitted, it retains Spanish for compatibility with existing clients. The portal always sends the selected language. This setting does not translate queries or modify the indexes or encoder weights.

Return to index ↑

CHAPTER 02 IATESIS: what it solves and where the data comes from

The project applies information retrieval, machine learning and generative AI to bibliographic data and thesis fragments. A catalog is a part of a library's data: describes documents by title, authors, year, subjects and links. A PDF provides text from the document. That is why the main presentation talks about AI applied to library data and thesis , and the search engine distinguishes both collections.

IATESIS is part of the applied research of NOETILUM . The project explores the processing of bibliographic data, the training of a neural encoder and the retrieval of information with sources. This demonstration allows clients to learn about the process, consult documents and review their evidence. Its author is Víctor Manuel Osornio Luna. The project uses the bibliographic source sectesis.txt; From it come the technical identifier SECTESIS , the IDs sectesis:…, the routes /api/v1/sectesis and the variables SECTESIS_*. These technical names are retained to maintain compatibility of existing data, API, and installations. SECTESIS identifies that source and its previous implementation; It is not an AI technique nor the current display name of the project. The project does not implement all the processes of a library: circulation, acquisitions and user administration are left out of this search engine.

The expected result for a person is to find references or passages, open their sources and, optionally, ask for an essay based on them. For a library, it provides a reusable index, access to extracted text, and traceability by document and page. The fact that you find results does not in itself prove that they are the most relevant.

Return to index ↑

CHAPTER 03 593,067 and 12,000 are figures from different stages
verified figure What counts What it doesn't mean
593,067 records Complete catalog prepared from this TXT, indexed with BM25 and vectorized There are not 593,067 PDFs nor the entire UNAM collection
12,000 records Reproducible selection to prepare for encoder training It is not the size of the searchable catalog
11,010 titles 12,000 minus 988 titles with less than four useful tokens and 2 normalized duplicates Not everyone updates the weights
8,750 titles Training partition: yes calculate gradients Did not train with validation and test titles
1,144 titles Validation to select the era They are not relevant human judgments
1,116 titles Reserved test partition Its existence does not imply that there is a human evaluation carried out
50 PDF / 49 indexed Reviewed files / files that produced fragments FAQS1.pdf did not produce extractable text
5,034 / 4,727 pages Physical page views / pages that produced fragments No additional OCR was run
6,541 fragments Indexed and vectorized PDF text units They are not different theses

Of the PDFs, 17 were left with status text_extracted, 32 with partial_text and one with no_extractable_text. There were 202 pages without text and 105 with fewer than 25 letters: those 307 pages did not generate fragments. Just because a PDF is indexed does not mean that all its content has been interpreted.

The file retains the historical name sample.jsonl, but in this delivery contains 593,067 records because it was prepared with --sample 0. The encoder training was executed on this server: four epochs, with selection of the third due to validation loss. It is not appropriate to describe it as a laboratory with 12,000 records simply imported.

Return to index ↑

CHAPTER 04 Two preparation routes and one way to query
  1. 01

    Read ALEPH

    Identifier, labels and subfields of the record.

  2. 02

    Prepare JSONL

    593,067 records; normalized and missing fields visible.

  3. 03

    Index words

    SQLite FTS5 organizes terms and records for BM25.

  4. 04

    Train encoder

    12,000 selected; 8,750 titles update the weights.

  5. 05

    Calculate vectors

    Fixed weights → 593,067 vectors; IDs in the same order.

The query selects catalog or PDF ; Hybrid search combines two methods within the chosen collection. It does not automatically join an catalog record and a PDF file or search both collections simultaneously.

Return to index ↑

CHAPTER 05 TXT Path: from ALEPH to JSONL and SQLite

Input and objective: understand a record before computing numbers

The entry is sectesis.txt , a sequential ALEPH bibliographic export of 866,475,615 bytes. It contains files that describe theses, not the transcription of all of them. The objective of this stage is to convert these lines into records with consistent fields and preserve their provenance. No Llama intervenes nor are weights learned.

records() reads strict UTF-8 and groups consecutive lines by the nine-digit identifier. The format requires that ID, a five-position label, the letter L, and the content; $$ separates subfields. An uninterpreted line is logged as a failure when a log is already in progress. The implementation assumes that the lines of a token are contiguous. Actual Parser: ingest.py , lines 8–35 .

LINE = re.compile(r"^(\d{9}) (.{5}) L ?(.*)$")
SUB = re.compile(r"\$\$([^$])([^$]*)")

Actual TXT fragment:

000000001 1001  L $$aAceves Ramirez, Noe de Jesus$$esustentante
000000001 24510 L $$aOrigen, causas, evolución, perspectivas y soluciones propuestas al problema habitacional en México /$$ctesis que para obtener el título de Arquitecto, presenta Noe de Jesus Aceves Ramirez
000000001 502   L $$bLicenciatura en Arquitectura$$cUniversidad Nacional Autónoma de México,$$d1979$$gFacultad de Arquitectura,

canonical() transforms the tags into an object with known fields. The following priorities are software rules, not human fixes:

Output field Rule applied Example or limitation
id Prefix sectesis: + ID ALEPH sectesis:000000001 ; It is not the hash of a thesis
title First 245$a , followed by 245$b if it exists Retains the title; does not automatically incorporate all responsibility 245$c
authors / contributors 100$a / 700$a Absent names are not invented
degree 502$b Licenciatura en Arquitectura
institution 502$c ; if missing, 710$a explicit fallback
faculty 502$g ; if missing, 710$b explicit fallback
year First 264$c , then 260$c , then 502$d ; extracts the first four-digit year It also preserves year_raw ; an extracted date may require review
subjects Subfields of 600, 610, 611, 630, 650, 651 and 653 Does not generate new subjects
abstract / urls 520$a / 856$u They are catalog fields; a link is not equivalent to having read your document

Verbatim rules: canonical() , lines 37–68 .

Selection of fields from the actual canonical record—the full object contains more—:

{
  "id": "sectesis:000000001",
  "source_id": "000000001",
  "title": "Origen, causas, evolución, perspectivas y soluciones propuestas al problema habitacional en México",
  "authors": ["Aceves Ramirez, Noe de Jesus"],
  "year": 1979,
  "faculty": "Facultad de Arquitectura",
  "degree": "Licenciatura en Arquitectura"
}

Three different normalization operations

Operation What does it really do? What is it used for?
clean() Unicode NFC, spaces following one, trim and end signs /,;: Clean the presentation of fields without deleting accents
fold() casefold, Unicode NFKD and combining mark removal Compare without distinguishing capital letters/accents; preserves punctuation and words
tokens() Applies fold , extracts \w+ sequences, removes units of two characters or less and a fixed list of words; keep the first 64 TF-IDF input and lexical query normalization

The training deduplication key is fold(título) , not a list of tokens. Two titles that differ only by score can still be different by that rule. There is no lemmatization—reduction to a dictionary form—nor identification of semantic duplicates. Cleaning and key: ingest.py , 10–13 ; project tokens: search.py, 6–7 .

Complete preparation, quality control and output

prepare() writes one object per line to sample.jsonl and the profile to profile.json . The basic CLI has --sample 12000 by default; This delivery retains the entire TXT , as evidenced by sample_method=full_corpus and sample_size=593067. --sample 0 means complete corpus. A positive sample uses reservoir sampling - it goes through the entire origin and keeps a uniform selection, not the first few rows. The profile is also calculated on all the read origin. Preparation: ingest.py , 85–117 .

Parameter or control Applied Default or limit Function
JSONL Scope 593,067 records Basic CLI --sample 12000 ; 0 retains all Separate learning sample catalog size
Preparation seed 20260909 Same defect in CLI Plays a selection when sampling is used and the order of the origin is maintained
Year for review 2026 Basic CLI 2026; supported range 1500…9999 Marks years out of 1500…year of validation; does not correct them
Format alephseq_v2, strict UTF-8 Unknown initial format is rejected Identify the rules used to interpret the data
Existing output It is not overwritten Require new output if JSONL/profile/temp already exists Avoid mixing versions
Integrity Source SHA-256 and JSONL; checks size/date of source during read Code controls Detects changes or mixtures; does not demonstrate bibliographic quality

The real profile reports 13 records without title, 17 without author, 1,828 without year, 31,687 without faculty and 296 without degree . Marks 1,827 years for revision and 87 lines not interpreted. Only 2 records have a summary according to the parser; 193,503 have subjects. These are presence measures and automatic flags, not a catalog certification. Incomplete records remain in the catalog; another filter then decides which ones are suitable for training. Evidence: payload/sectesis/profile.json of delivery.

Each record contains two texts prepared for different uses. Literal fragment of canonical():

rec['semantic_text']='\n'.join(v for v in [title, '; '.join(subjects),abstract] if v)
rec['lexical_text']='\n'.join(v for v in [title,'; '.join(rec['authors']),'; '.join(subjects),degree,faculty,year_raw] if v)

lexical_text is used for BM25. Although there is semantic_text , the active teaching encoder is titled , not the entire field. record_sha256 identifies the parsed fields of the record; is different from the byte hash of the TXT and the bibliographic ID.

From JSONL to SQLite: saving and indexing is not training

search.build() inserts each record into records , with ID, scope, year, faculty, and JSON. Creates docs , an FTS5 virtual table for the inverted index, and manifest , with version, count, and JSONL footprint. SQLite organizes text and terms for retrieval; it does not learn the neural matrix. Exact construction: search.py, 13–30 .

con.execute('CREATE TABLE records(id TEXT PRIMARY KEY,tenant TEXT NOT NULL,year INTEGER,faculty TEXT,payload TEXT NOT NULL)')
con.execute("CREATE VIRTUAL TABLE docs USING fts5(id UNINDEXED,title,body,tokenize='unicode61 remove_diacritics 2')")

FTS5 uses its own tokenizer unicode61. When constructing docs , the entire document text is saved: each document is not trimmed to the 64 encoder tokens. The ID is not indexed as text; It serves to link the result with its record. The actual output is sectesis/catalog.sqlite . The procedure rejects an existing route and publishes the temporary file when finished, to preserve a consistent version.

Return to index ↑

CHAPTER 06 PDF path: from file to fragments, IDs and index

What is a fragment and why is it used?

A chunk , or chunk , is a chunk of text extracted from a page that is saved as a search unit. This is text from the document, not a summary written by Llama or a new PDF file. A page can produce several fragments and a thesis many; That is why 6,541 fragments are not equivalent to 6,541 theses.

The text is divided to retrieve more precise passages and send only relevant material to the generation, rather than a complete thesis. Each fragment preserves the file, the page and their positions to return to the source. Dividing also has a cost: it may separate ideas or leave an explanation outside the passage; so some shared text is preserved between consecutive fragments.

How they were created in this project

PyMuPDF opens each PDF and extracts the available text per page with page.get_text("text", sort=True) . Spaces and control characters are normalized. No generated summaries are added or OCR is run. If the PDF already contains a faulty text layer, that extraction may preserve errors; a scanned PDF without text may not provide fragments.

Pages are divided into chunks of up to 1,600 characters , with a target overlap of 200 characters , adjusted to word limits. No pages are mixed. A minimum of 25 letters is required on the page. The offsets char_start and char_end refer to the normalized text on that page; the final end is exclusive.

Real example from 0609550.pdf, physical page 33:

Realizar la planeación, programación y evaluación periódica de las actividades que se efectúan mediante el análisis de la situación de las diferentes áreas.

is a shorthand : it corresponds to positions 562–718 within a 1,596-character fragment. It is not the full text of the fragment. Actual metadata:

{
  "id": "pdf-61d5c568d55a2a6b6620dcc56b7eaae1925b3241837a5180719e3561681a611d:p33:c0",
  "doc_id": "pdf-61d5c568d55a2a6b6620dcc56b7eaae1925b3241837a5180719e3561681a611d",
  "file": "0609550.pdf",
  "page": 33,
  "char_start": 0,
  "char_end": 1596
}

doc_id concatenates pdf- and the SHA-256 of the original bytes. p33 identifies the physical page; c0 the first fragment of that page, numbered from zero. A change of bytes changes the identity of the document. With equal bytes, extractor and options, the segmentation is reproducible. The TXT example and this PDF are different documents; no relationship is being asserted between them.

Two representations of the same set are produced: pdf/chunks.jsonl facilitates inspection and exchange and pdf/catalog.sqlite allows searching. In SQLite, files saves documents, chunks saves IDs and pages, and fts indexes the text in fragments. A manifest.json logs skip files, counts, options, and hashes. The originals remain at data/PDF; They are not converted to weights or replaced by SQLite.

Two real consecutive fragments

In 0609550.pdf, physical page 33 produced these two units, checked directly in SQLite. They share the same doc_id shown above:

Fragment Home included Excluded ending Saved characters
…:p33:c0 0 1.596 1.596
…:p33:c1 1.396 1.759 363

The 200-character stripe 1,396–1,596 appears in both. The second is shorter because it reaches the end of the text on that page. These positions count normalized text characters, not PDF bytes or Llama tokens. On other pages, adjusting the word limit can change the effective length and overlap.

The code looks for a space so as not to cut a word when it reaches the target size, saves the section and backtracks to the overlap area to start the next one. It does not automatically detect chapters or guarantee cuts for complete sentences. Each page is processed separately; no text from the previous page is repeated within the next.

Each drive is written to chunks.jsonl and SQLite with the same ID. BM25 indexes your text; vectorization applies the fixed encoder and saves a row of 1,024 values ​​associated with that ID. The search returns the fragment with its page and link to the original. The active encoder only uses up to 64 useful tokens, so a fragment may be completely available for BM25 and human reading, but partially represented in dense search.

Code: extract, chunks and index_pdfs .

Applied parameters, justification and observed coverage

The size 1,600 and the overlap 200 are decisions of this implementation: they delimit the passage and preserve some continuity. No comparison was made to show that are optimal. The pdf/manifest.json file in the delivery confirms these options and PyMuPDF 1.26.7.

Extraction parameter Applied value What is it for? How can it be changed
chunk_chars 1,600 characters Limit the size of each search unit CLI --chunk-chars, between 200 and 8,000
overlap_chars 200 target characters Share context between consecutive fragments of a page CLI --overlap-chars, from 0 to less than half of chunk_chars
min_letters 25 letters per page Skips pages with very little alphabetic text Python object Options ; no current CLI flag
max_file_bytes 67,108,864 bytes, 64 MiB Reject files that exceed that input limit Options
max_pages 2,000 pages per PDF Limit extraction by file Options
max_page_chars 200,000 characters extracted per page Rejects excessive extraction before normalizing Options
previous Reuse of 49 PDFs in the final build Avoid repeating extraction of identical content CLI --previous, with the same versions and options

Definition and validation: Options and validate . Options actually exposed by the command: main: index and vectorize . A value in Python code is not automatically equivalent to an option available on the screen or in Compose.

This fragment literal of the algorithm shows the cut. Look for the last space in the second half of the section to avoid breaking a word. The code following this fragment also adjusts the beginning of the next unit:

end = min(start + size, len(text))
if end < len(text):
    boundary = text.rfind(" ", start + size // 2, end)
    if boundary > start:
        end = boundary
yield start, end, text[start:end]
if end == len(text):
    break
next_start = max(start + 1, end - overlap)

Source: chunks, lines 49–67 . Normalization uses NFC and collapses spaces; preserves capital letters, accents and hyphenated words. It does not stemmize or automatically recompose words cut at the end of a line: normalize .

In the final delivery, the index files were reconstructed by reusing the previous extraction of 49 documents, according to files_reused=49. Those 49 PDFs were not extracted again during that build. For reuse, index_pdfs requires the same extractor/options fingerprint and SHA-256 from each original; then write the new SQLite and JSONL. Vectorization does apply the delivery encoder's fixed weights to those fragments. Code: index_pdfs: reuse by footprint and vectorize: inference with existing weights .

Registered status Files Correct reading of the result
text_extracted 17 All of its pages passed automatic text checks; human fidelity needs to be valued
partial_text 32 Some pages were left without fragments due to the absence or scarcity of text
no_extractable_text 1 FAQS1.pdf produced no fragments
Total 50 49 contribute at least one fragment

The manifest totals 202 pages without text and 105 with less than 25 letters : 307 of the 5,034 pages did not originate fragments. The remaining 4,727 produced 6,541 units. Therefore, "49 indexed PDFs" means partial or total coverage of the extractable text of those files, not guaranteed full reading. Titles obtained from PDF metadata are also not validated bibliographic metadata. Source of rules and states: extract: extraction by page and states ; Evidence of this run: pdf/manifest.json included in the Release of artifacts .

The indexer ignores symbolic links, duplicate content, and files that exceed limits; respects the permission to copy text and does not open files that require a password. Verify that the original does not change during extraction and publish to a new destination. These checks protect data consistency; they do not interpret images, tables or formulas: index_pdfs: controls and new index .

Return to index ↑

CHAPTER 07 What is BM25 in this project

BM25 —Best Matching 25— is a lexical ranking method: orders matches between words in the query and indexed text. It was chosen as a retrieval reference implementable without neural training and capable of searching for terms present in the data. Its existence in the laboratory allows us to compare dense search with a real lexical method; It does not prove that one is superior in all questions.

What goes into the index and what goes into the query

The catalog saves title and body in the FTS5 virtual table docs; body contains title, authors, subjects, degree, faculty and original year. The ID is UNINDEXED: it is used to link the result with its token. PDF uses table fts , with the full text of each fragment and its ID unindexed. They both declare tokenize='unicode61 remove_diacritics 2' . Applied code: build: catalog columns and loading and index_pdfs: PDF schema .

There are two different tokenizers. FTS5 indexes all text that is saved in its columns. The Python function tokens() prepares the query and also the encoder inputs: normalizes capitalization and accents, discards stop words and terms of two characters or less, and preserves the first 64 useful terms. Therefore, the 64 limit does not trim the text stored in the BM25 index. Stemming, lemmatization, and synonym expansion were not applied.

Literal code of tokens, with its list of excluded words:

STOP=set('de la el los las en y a un una para por con del al que sobre entre como su se es tesis estudio analisis'.split())
def tokens(s):return [w for w in re.findall(r'\w+',fold(s)) if len(w)>2 and w not in STOP][:64]

Sources: tokens, lines 6–7 and fold: Unicode normalization . Teaching example: El problema habitacional en México produces problema , habitacional , mexico . The code forms "problema" OR "habitacional" OR "mexico" ; just match part of the query. The screen does not offer advanced phrase, AND, or synonym syntax although FTS5 has additional capabilities.

How the score is calculated and ordered

The contribution of each term depends on its frequency in the document, its rarity in the collection, and the length of the document. Repetition becomes saturated: a hundred appearances do not provide a hundred times more usefulness. To understand the positive calculation shown by this API:

aporte = IDF × f × (k1 + 1) / (f + k1 × (1 - b + b × longitud / longitud_media))
score = suma de aportes de los términos consultados

f is column-weighted frequency; IDF expresses rarity, and longitud counts FTS tokens. FTS5 internally sets k1=1.2 and b=0.75 ; They are not arguments of this search engine. SQLite returns the reverse sign to sort from least to greatest and our API changes it on publishing. The exact definition of FTS5 specifies its IDF and column weighting.

Parameter or decision Catalog PDF Status on this project
Scoring function bm25(docs,0,2,1) bm25(fts) Applied in SQL
Column weight ID 0; title 2; body 1 Text with default weight 1 Fixed in code; not learned
Query operator OR between cited tokens Equal Fixed in code
Results top_k 5 by default; API 1–50 Equal Configurable by request
Scope tenant fixed on server Configured PDF folder Independent collections
Filters Inclusive years and exact normalized faculty doc_id exact They are applied before choosing results
Tie-breaker Upstream ID Upstream fragment ID Deterministic for those candidates

The title of the catalog participates in its column and also in body; Do not interpret weight 2 as a guarantee of "double relevance" of a thesis. Year filters exclude records without a year when they cannot satisfy the condition. The faculty is compared completely after normalization, not by partial coincidence. PDF does not have year or faculty filters in this API. Sources: LexicalIndex.search , Engine.lexical and Query: API Limits .

This SQL is an equivalent presentation of the implemented query, with illustrative values ​​and no optional filters:

SELECT r.payload, bm25(docs, 0, 2, 1)
FROM docs JOIN records r ON r.id = docs.id
WHERE docs MATCH ? AND r.tenant = ?
ORDER BY bm25(docs, 0, 2, 1), r.id
LIMIT ?;
-- Parámetros: '"problema" OR "habitacional" OR "mexico"', 'global_unam', 5

LexicalIndex.search adds filters using SQL parameters; Llama does not write or execute that query. The index looks for matches in existing data: just because a record includes “Mexico” does not prove that it answers the entire question. The BM25 score is neither a probability nor a percentage of truth and should not be added directly to the cosine of another method.

What could be compared in research

Remaining as proposed experiments are: OR versus all-term matching, other title/body weights, and linguistic normalization. They require code changes or new indexes, revised relevance cases and comparison with this fixed configuration. Those variants were not run nor were the current values ​​chosen through a full catalog relevance optimization.

Return to index ↑

CHAPTER 08 How the encoder is trained

What does it mean to train in this laboratory

Training consists of modifying an array of numbers, W , to reduce a loss function. The input is the TF-IDF representation of a title; the output is a 1,024 coordinate vector. The model learns to approximate two incomplete versions of the same title and distinguish them from other titles in the batch. It doesn't learn from questions answered by librarians or from PDF paragraphs. It is a didactic linear network built from scratch, without pre-trained linguistic knowledge.

The historical process was recorded in BUILD_REPORT.json, sectesis/neural/encoder.json, training_sample.jsonl and splits.json. The kit declares training.performed=true, four epochs and the third as selected. In this revision of the manual those files were verified; training was not repeated. The ingestion and encoder code retains the same traces as the build of commit 9193ce88f586783d3d8ab5dd93ec6880f49116a9 .

Step 1: Choose data, filter and separate partitions

reservoir() loops through the entire JSONL and selects 12,000 records with seed 20260909 . Each registration has the same opportunity to enter, but balance by faculty or year is not guaranteed. The input order also matters for playing the sample. Selection: ia_artefactos.py , 81–116 .

The real sample was retold applying the rules of the program, without training:

Operation Proven result What is it for?
Initial selection 12,000 records Limit the cost of learning
Exclude titles with less than four useful tokens 988 excluded; 11,012 eligible Require some content to create two views
Keep the first of each fold(título) 2 duplicates excluded; 11,010 unique titles Avoid repeating the same key between partitions
Training partition 8,750 titles Adjust TF-IDF and update W
Validation partition 1,144 titles Choose the time for loss
Test partition 1,116 titles Reserved; does not participate in adjustment or selection

The recalculated membership and order match splits.json . The filter and deduplication run in that order. Key fold(título) preserves punctuation; It does not automatically group similar titles or all documents on the same topic. Filter and partitions: neural.py , 43–50 .

The split_of() function takes the first eight hexadecimal characters of the SHA-256 of that key and applies modulo ten. It's a rough 80/10/10 rule, not three exact percentages or one split per year. It does not use the sampling seed. Code: neural.py , 20–22 .

n=int(hashlib.sha256(key.encode()).hexdigest()[:8],16)%10
return 'test' if n==0 else ('validation' if n==1 else 'train')

Step 2: Convert title to TF-IDF

TF-IDF assigns a value to each vocabulary term: consider how much it appears in the title and how many training titles it appears in. A vector of 4,096 positions uses the same order of terms for all texts. A zero position indicates that that term does not provide a signal at that input; It doesn't mean that the entire title is irrelevant.

The project uses sublinear frequency: for a present term, tf = 1 + ln(frecuencia). The smoothed IDF is ln((1+n)/(1+df)) + 1, where n=8750 and df count training titles that contain the term. Multiply both values ​​and normalize the vector to length L2 one. Words outside the vocabulary are omitted. Formulas from scikit-learn 1.8 .

Literal fragment: fit_transform learns vocabulary and IDF only with training ; transform applies those same values ​​to validation. Code: neural.py , 49–50 .

v=TfidfVectorizer(max_features=4096,min_df=2,tokenizer=tokens,token_pattern=None,lowercase=False,sublinear_tf=True,dtype=np.float32)
x=v.fit_transform([d['title'] for d in splits['train']]); xv=v.transform([d['title'] for d in splits['validation']])
Representation parameter Applied value and origin Effect and limit
max_features 4096, set in code; real vocabulary of 4096 Limit coverage and memory
min_df 2, set in code A term must appear in at least two train titles
tokenizer Project tokens() Removes accents/capitalization, short units and fixed list; maximum 64 useful tokens
token_pattern / lowercase None / False, explicit Delegate this normalization to the own tokenizer
sublinear_tf True, explicit Reduces the effect of repeating terms
dtype float32, explicit Four bytes per value; finite precision rough calculation
ngram_range (1,1), scikit-learn default Unigrams; non-characteristics of word pairs
norm, use_idf, smooth_idf 'l2', True, True, scikit-learn defects Normalization and smoothing IDF
max_df, binary 1.0, False, scikit-learn defects Does not exclude for maximum frequency less than 100% nor does it convert all frequencies to one

The defaults correspond to scikit-learn 1.8.0 , version registered in the build. They were distinguished from the arguments that the project expressly establishes. Signature of TfidfVectorizer 1.8 . In this audit the 4,096 saved IDFs match the train calculation within a maximum error of 5,69 × 10⁻⁷ , compatible with float32.

Step 3: Transform, compare and adjust the weights

The network has a layer 4096 → 1024 with no bias . It is a multiplication by the W matrix, followed by L2 normalization; it has no attention blocks or Transformer architecture. Class Encoder , neural.py , 16–19 .

class Encoder(nn.Module):
    def __init__(self,n:int,dim:int=1024):
        super().__init__();self.linear=nn.Linear(n,dim,bias=False)
    def forward(self,x):return nn.functional.normalize(self.linear(x),p=2,dim=1)

For each batch of up to 96 titles, the software creates two independent masks that hide each feature with probability 0.15. It doesn't remove exactly 15% of words from each title. The two views of the same title are the positive pair ; the other titles in the lot act as negative , without a person having judged their thematic relationship.

aa=a*(torch.rand_like(a)>.15); bb=a*(torch.rand_like(a)>.15)
z1=model(aa);z2=model(bb);sim=z1@z2.T/.1; labels=torch.arange(len(a))
return .5*(nn.functional.cross_entropy(sim,labels)+nn.functional.cross_entropy(sim.T,labels))

This fragment comes from loss_batch() , lines 53–58 . a has up to 96 rows for 4,096 entries. z1 and z2 are up to 96 by 1,024. The sim matrix compares each row in one view with all rows in the other; its diagonal contains the positive pairs. 0.1 temperature scales those similarities before the loss. Cross entropy penalizes another row receiving more support than the corresponding one; It is calculated in both directions and averaged. It is not a probability of relevance of a thesis.

The update is executed with this literal line. Actual loop: neural.py , 59–66 .

a=torch.from_numpy(x[ix].toarray());opt.zero_grad();loss=loss_batch(a);loss.backward();opt.step();losses.append(loss.item())

Your step-by-step read: convert batch to numbers for PyTorch; delete previous gradients; calculate loss; backward() calculates how each weight influences that loss ; step() modify the weights with AdamW ; save the loss value. The network learns a shared transformation, not a weight per thesis.

Learning parameter Applied/origin What is it for?
Dimension 1024, fixed in the network Output vector size
Bias bias=False, explicit Does not add an independent vector to W @ x
Learned parameters 4.194.304 = 1024 × 4096 W coefficients modified by gradients
Optimizer AdamW Calculate updates from gradients
lr Rate 0.002, explicit Update step scale
weight_decay 0.001, explicit Separate decay of weights during optimization
betas / eps (0.9,0.999) / 1e-8, PyTorch 2.10 defects Averages of the gradient and its square; numerical stability
Lot size Up to 96, set in code Memory per step and number of negatives
Concealment Probability 0.15, fixed Two incomplete views of each entry
Contrastive temperature 0.1, fixed Scale of comparisons within loss
Eras 4, option --epochs ; range 1…100 Training tours
Seed 20260909 Initialization, masks and training mix
PyTorch Threads 2, fixed in code Parallelism of this CPU execution

AdamW construction and controls: neural.py, 39–58 . The betas and epsilon are defaults inherited from the version, not values ​​selected through experiments of this project. AdamW 2.10 . The output normalization divides by the maximum of the norm and 1e-12 , PyTorch's default epsilon; an exactly zero vector remains zero. L2 Normalization .

The function of each value is explained; its optimality is not proven. No hyperparameter search was found that justified 4096, 1024, 96 or rates as the best values ​​for the corpus. Varying them is a line of research. The current CLI does not offer --dimension , --learning-rate , or --temperature for this encoder: changing those constants requires a code version and new artifacts.

Step 4: Validate and preserve an epoch

Each epoch mixes only the 8,750 training titles. The validation uses 1,144 titles, without gradients, with reproducible masks using seed + 77. The state with the least validation loss is saved and is recovered after completing the four epochs . The training was not stopped in the third nor was it chosen using a test. Validation and selection: neural.py , 67–87 .

Time Training loss Validation loss
1 0,136437 0,153787
2 0,109401 0,154933
3, selected 0,098081 0,152873
4 0,095806 0,152960

The loss measures the contrastive objective chosen, not the percentage of success or user satisfaction . The code averages batch losses without weighting them by size. In this partition there are 92 train batches per epoch – the last one with 14 titles – and 12 validation batches – the last one with 88 titles. Changing lot or objective modifies the interpretation of the loss; These numbers should not be compared as a universal scale of quality.

encoder.json records training_seconds=21.916 - The measured interval starts before preparing the sample in train() and ends when creating metadata. It does not include the subsequent vectorization of the 593,067 records, PDF ingestion or complete delivery. It is a historical reading of a run, not a repeated benchmark. Python 3.11.16, PyTorch 2.10.0+cpu, NumPy 2.3.5, and scikit-learn 1.8.0 were used.

What can be configured and what is reused

The kit accepts --train-sample 12000 —limit 20…50,000—, --epochs 4 and --seed 20260909. The direct training function rejects more than 50,000 records; That is why the kit trains a sample and then encodes the entire corpus. Since --txt requires --retrain ; with --catalog and without --retrain checks and reuses existing weights. Options: ia_artefactos.py , 491–503 ; train/reuse decision: 299–333 .

The historic delivery did train the title encoder. This transformation was then set for the entire catalog and the PDFs. Page queries do not update their weights. They don't train Llama either.

Return to index ↑

CHAPTER 09 From text to numbers: real example of weights and vectors

Follow a record from the text to its number row

The actual title sectesis:000000001, displayed in the TXT path, produces nine non-null TF-IDF features. Some are causas = 0.345828, evolucion = 0.310049, habitacional = 0.407594 and mexico = 0.158565. The vector has 4,096 positions; the others can be zero. These values ​​describe the input calculated for that text - they are not the parameters learned from the linear layer.

The following notation explains the operation; It is not a new training nor a transcription of a function from the repository:

x = TF-IDF(título)               # 4096 valores
z = W @ x                       # W: 1024 filas × 4096 columnas
embedding = z / max(||z||₂, ε)   # 1024 valores; ε = 1e-12

@ represents matrix multiplication by vector. Each output coordinate combines many inputs with different weights. L2 measures the numerical length of the vector; normalizing makes it easier to compare your orientation. A coordinate is not an identified human word or category.

First six weights of row 0 of the real matrix W, rounded:

[0.032730, -0.009829, 0.007284, 0.007827, 0.037297, -0.044680, ...]

First six values ​​of the vector saved for sectesis:000000001:

[0.053612, -0.016303, 0.021445, 0.016783, 0.001762, -0.007057, ...]

First six values ​​of the PDF fragment vector …:p33:c0, row 906 of the PDF vectors:

[-0.003362, 0.006055, 0.056237, 0.017407, -0.037716, 0.040301, ...]

They are real numerical extracts, not made-up matrices or complete weight files. In the check saved as artifacts/ejemplo-encoder-real.json, re-encoding the title played the stored vector within numerical tolerance. The PDF example belongs to another document: there is no automatic correspondence between that thesis and the TXT file.

Weights, metadata, IDs and vectors serve different functions

Archive What does it keep How to use
weights.pt State of the learned linear array Apply the same transformation to titles, questions and fragments
encoder.json Vocabulary and IDF, dimension, history, summary partitions, seed and traces Reconstruct preprocessing and document provenance
splits.json IDs grouped in train, validation and test Allows you to audit who participated in each stage
training_sample.jsonl The 12,000 records selected before the filter It allows us to reconstruct the historical entry to learning
vectors.npy Numerical output of encoding each text Allows you to search without recoding all documents by query
ids.json ID corresponding to each row of vectors.npy ids[906] tells which fragment vectors[906] is from
integrity.json Traces of files from the set Detects corruption or mixing; does not certify relevance of results

The kit separately preserves training_dataset_sha256, the fingerprint of the sample, and source_sha256, the fingerprint of the complete catalog vectorized . That the latter identifies 593,067 records does not mean that all of them trained the matrix. Provenance and stamping: ia_artefactos.py, 148–163 .

Vectorizing is inference with fixed weights

LocalEncoder rebuilds TF-IDF using the saved vocabulary and IDF, loads weights.pt into CPU with weights_only=True, and puts the network into evaluation mode. Your encode() method works in batches of 96 with torch.inference_mode(), without gradients or optimizer steps. Literal fragment of the inference path: neural.py, lines 27–38 .

with torch.inference_mode():
    for i in range(0,len(texts),batch_size):
        x=torch.from_numpy(self.vectorizer.transform(texts[i:i+batch_size]).toarray())
        out.append(self.model(x).numpy())

After training, train_and_encode() processes all 593,067 titles and writes a float32 matrix using open_memmap, producing a large file in blocks. Each batch supplies its rows and IDs in the same order; non-finite numbers and duplicate IDs are rejected. Building the complete matrix: ia_artefactos.py, 121–147.

The 6,541 PDF fragments pass through the same title encoder with fixed weights, as documented by training_performed=false from the PDF collection. The input text is now a fragment; it does not magically change the vocabulary or add knowledge to W . It is a transfer of representation that requires evaluating its quality in that new type of text. PDF Orchestration: ia_artefactos.py, 334–347 .

Matrix delivered Shape Stored values
Catalog (593067, 1024) 593,067 rows × 1,024 float32 numbers
PDF (6541, 1024) 6,541 rows × 1,024 float32 numbers

The catalog values ​​alone occupy approximately 2.43 GB decimal ( 593067 × 1024 × 4 bytes), plus the small NPY header. It is data size, not additional RAM measured per query or weight size. The W array has 4,194,304 parameters; the document matrix has many more rows because it saves its results.

Visible limits of this representation

The tokenizer retains at most 64 useful tokens per text . A 1,600-character snippet can exceed that limit: dense representation does not necessarily incorporate all of its content. BM25 and human reading do have the stored text of the fragment. Terms outside the vocabulary do not provide input coordinates either.

The delivery reports 7,087 title vectors and 146 PDF fragments with a standard lower than 1e-8 , the kit's counting criterion. Texts with no signal in the vocabulary can produce zero vectors; They should not be interpreted as "worthless thesis." They remain in the lexical collection. Investigating vocabulary, truncation, and encoder alternatives can improve coverage, but that improvement must still be measured with queries and relevance judgments.

Return to index ↑

CHAPTER 10 Query, dense, hybrid search and RAG

How to transform a question into dense candidates

Dense search applies the same encoder that created the collection vectors to the query and compares numbers in the same 1,024-dimensional space. For two vectors q and d, the dot product sums q[i] × d[i] at all coordinates. With L2 norm equal to 1, this product coincides with the cosine similarity. The model normalizes its outputs; a zero vector preserves zero norm and does not provide a positive dense match.

exact block search is used, without FAISS, ANN or GPU. It is an inspectable implementation: it loops through the eligible vectors and keeps the best ones. Its cost increases with the number of records and dimensions; Blocks limit intermediate work, but do not eliminate travel. Code: DenseIndex.search and Engine.search .

Parameter or step Applied value Function and limit
Dimension 1.024 Must match between query, weights and matrix
Storage float32, vectors.npy Four bytes per coordinate
Reading mmap_mode='r', allow_pickle=False Memory-accessed disk array; does not modify vectors
Block size 4,096 rows Process groups and gather their best candidates
Publication threshold score > 0 Rule out non-positive similarities; It's not calibrated trust.
Standards Check Zero less than 1e-8 or close to 1 with atol=1e-4 Detect invalid arrays before using them
final order Descending score, tiebreaker by ID Sorting within the applied representation
Final quantity top_k=5 default; 1–50 in API Maximum number of sources, not LLM sampling settings

When loading, hashes, aligned IDs, shape, numeric type and finite values ​​are checked. The first search may include expensive reading and verification. The array is opened with mmap , but the catalog loads its metadata in a Python list and the IDs remain in RAM: not all the browser works without taking up memory. Code: DenseIndex. init and Engine.load_dense .

In the catalog, the records allowed by field, years and faculty are chosen before multiplying. The following extract literal shows the product and retention per block:

scores = self.vectors[allowed] @ q
for j in np.argsort(-scores, kind='stable')[:k]:
    if scores[j] > 0:
        candidates.append((float(scores[j]), allowed[j]))

Source: DenseIndex.search: product and candidates . PDF calculates the block product and masks fragments other than doc_id before selecting results. A query without useful representation may retrieve nothing in dense and still find matches by BM25. Nor does a high cosine prove that a fragment answers the question: that statement needs evaluation.

How the two lists are merged with RRF

The hybrid search retrieves up to 50 BM25 and 50 dense candidates within the chosen collection . RRF—reciprocal rank fusion—adds one contribution per position; ranges start at 1:

RRF(documento) = suma, por cada lista donde aparece, de 1 / (60 + rango)

A document absent from a list does not count for it. Teaching example, does not measure score : positions 1 and 3 give 1/61 + 1/63 ≈ 0,032266. This allows orders to be combined without directly mixing BM25 and cosine. Both lists have the same weight; the constant 60 and depth 50 are fixed in the services, not in the form.

Literal code that avoids adding the same ID twice to a list and preserves the original signals:

for rank,d in enumerate(ranking,1):
    if d['id'] in used:continue
    used.add(d['id']);scores[d['id']]+=1/(constant+rank);docs[d['id']]=d
    signals[d['id']].append({'method':d.get('retrieval_method'),'rank':rank,'score':d.get('score')})

Sources: rrf, lines 49–57 , Service.search: depth of lists and Engine.search: PDF merger . Contributions are then sorted and the first top_k are returned. In PDF, several fragments of a thesis can occupy different positions: diversification by document was not applied. Hybrid also does not search both TXT and PDF simultaneously: it merges two methods within the selected collection. Combination is an implemented technique; its relevance improvement is not demonstrated for this entire delivery.

What the server receives and what Llama receives

Catalog and PDF paths accept message between 2 and 2,000 characters, method ( bm25 , dense , hybrid ), top_k 1–50, and generate false by default. The catalog admits years and faculty; PDF supports doc_id. They reject unknown fields and bodies larger than 32 KiB. Queries for each collection have two simultaneous slots per process; if they get busy they return HTTP 429. This is concurrency control, not a certified multi-user capability. Code: Query and BodyLimit , PDFQuery and install and Service: slots and response .

References or fragments are first displayed on the website using generate:false . If “Compose with Ollama” is selected, another request is then sent with generate:true ; the server again fetches and prepares context. The initial sources remain visible during the wait and are preserved if the drafting fails. This flow can be read in SectesisRequests.search .

RAG means retrieval-augmented generation: question → search → evidence → context sent to Llama → response with citations. For catalogues, limited metadata of up to ten references are sent; for PDF, up to eight fragments and up to 2,000 characters per extract. The interface asks for five results by default. Not all 593,067 records or all theses are sent to each question. Context construction: generate from catalog and generate from PDF .

Llama writes; the encoder prepares representations to search. The LLM is reused already trained and its weights were not adjusted with these theses . Questions also do not automatically train models. Citation checking checks for flags such as [1] and membership in the available set, not semantic support for each statement. When opening a PDF, the API compares its SHA with the index and rejects a modified version; This traceability allows the original to be reviewed, but does not replace the review: file: opening and fingerprint of the original .

Variants to study later

Comparing another similarity threshold, depth or RRF constant, different weights per method, clustering by thesis, or neighbor fragment recovery requires a new experiment. The index_pretrained adapter allows you to prepare catalog with a local SentenceTransformer, revision, and registered hashes; was not applied in these artifacts. The current PDF vectorization only accepts the educational encoder. ANN, rerankers and long context embeddings are possible extensions, not components implemented here. Existing option code: index_pretrained: local catalog adapter ; PDF constraint: vectorize: supported model .

Return to index ↑

CHAPTER 11 Evaluation: what was proven and how to measure relevance

Functioning, learning and quality are different evidence

Available evidence What allows us to affirm What it doesn't prove
Code testing with artificial data Rules, errors, contracts and controlled cases are verified Quality of the theses delivered
Loss of training and validation The encoder optimized its contrastive objective and an epoch was selected Percentage of correct chat responses
BUILD_REPORT.json delivery Both collections respond with all three methods and a PDF was opened Human relevance, Ollama response or concurrent capacity
HTTPS and browser checks Operation of the service published in verified queries Overall quality for any questions
Actual Ollama measurements Durations under registered conditions Scientific accuracy or speed guaranteed

The final build used two queries derived from each source , each with up to five of its first useful tokens. Requested three results with generate:false for BM25, dense and hybrid: six collection/method combinations, with two answers in each. The report records HTTP 200 and three citations in each response, plus opening an original PDF. The function requires extractive mode and some evidence by combination; it does not compare results against judgments of relevance. It is executed using ASGI TestClient in the same process, without Ollama inference. Sources: sample_queries: queries derived from source and functional_check: ASGI check ; results in BUILD_REPORT.json within the private Release .

Implemented ranking evaluator

evaluate compares recovery from catalog with known positives. A positive is an ID that the case file declares relevant to a question. Does not evaluate PDF or generation. It receives between 1 and 10,000 cases, verifies existing IDs in the scope and calculates metrics with k=10 by default, configurable between 1 and 50. An illustrative scheme—not a new validated judgment—is:

{
  "id": "pregunta-ejemplo",
  "query": "problema habitacional en México",
  "relevant_ids": ["sectesis:000000001"],
  "human_validated": false
}

Before each measurement he does a warm-up consultation. It then measures another sequential call to the search engine, without LLM or concurrency. Saves retrieved IDs, corpus/case hashes, metrics per question and averages. human_validation_declared only reflects what is declared in the JSON; it does not certify that a person reviewed the sources. Code: evaluate: case validation and execution ; available arguments: CLI evaluate .

To read the formulas, k is the maximum number of results evaluated, H the positives found among them, P the total number of known positives and r a position starting at 1:

Implemented metrics Formula or calculation What question does it answer?
Recall@k H / P What fraction of the known positives did you recover?
Precision@k H / k What proportion of the k locations correspond to known positives?
MRR@k 1 / r of the first positive; zero if missing How soon does the first positive reference appear?
DCG@k Add 1 / log2(r + 1) for each positive Do the positives appear in high places?
nDCG@k DCG / DCG_ideal How close was it to the best possible order of those positives?

The evaluator uses binary relevance and removes duplicate IDs before taking k; Precision@k keeps denominator k even if there are fewer results. Teaching example: with k=5 , three known positives and two recovered in positions 2 and 4, Recall is 2/3 , Precision 2/5 and MRR 1/2 . DCG is 1/log2(3)+1/log2(5) ; the ideal places the positive three at 1, 2, and 3. This example explains the calculation and is not a measurement of the project. Literal implementation: metrics, lines 11–18 .

hit=sum(i in positive for i in ids)
dcg=sum(1/math.log2(j+2) for j,i in enumerate(ids) if i in positive)
ideal=sum(1/math.log2(j+2) for j in range(min(k,len(positive))))

For times, the code saves p50 as the bottom center element of the sorted latencies and p95 as the ceil(0.95 × n) - 1 element, using indices from scratch. Those percentiles describe your hot sequential calls; they do not represent initial loading, generation or multiple users. It does not calculate confidence intervals or statistical significance. Source: evaluate: summary and limits .

What historical results can be cited

The document VALIDACION.md: historical proxy comparison records 100 proxy queries for known titles for the 12,000 record lab on September 10, with another sample hash. They include Recall@10/MRR@10/nDCG@10: BM25 1,000/0,995/0,996, dense 0,960/0,874/0,895 and hybrid 1,000/0,964/0,973. It is a stated historical comparison, not a new benchmark from the 593,067 catalog or the PDFs. Their queries retain words from titles and only have one known positive, so they favor lexical matches and do not exhaustively judge other useful documents.

A relevant human benchmark of the complete catalog or the fragments was not found in the current delivery. To demonstrate improvement, it is necessary to construct independent questions, review positives and negatives, set partitions and criteria before comparing variants, and also publish errors and limitations. The existing evaluator provides a starting point for the catalog; evaluating PDF and support of Llama claims needs to expand the protocol and tools.

test_pdf_extension.py tests create artificial PDFs and controlled weights; test_pdf_repository.py uses data/PDF if it exists and checks extraction, BM25, opening and reuse. Separating these types of evidence prevents presenting a fixture as a result of the theses. Code: test_pdf_extension: creation of test data and test_pdf_repository: integration with data/PDF .

Return to index ↑

CHAPTER 12 Llama: operation, parameters and role in the project

Llama is the drafting model; Ollama is the program that loads and executes it. The finder delivers a few references or passages to Llama using the local API /api/chat . The LLM weights remain fixed during the consultation. Training the title encoder, explained above, is a different process.

Llama 3 is an autoregressive Transformer: it calculates the next token from the previous ones and repeats that step until finished. The Instruct variant was prepared by Meta to follow instructions through supervised tuning and human feedback; that previous work was not done for this project. The original sheet mainly describes usage and evaluation in English, so the quality in Spanish needs to be checked with our questions. Official file for Meta Llama 3 .

Actually installed model sheet

Local reading from September 14, 2026 using /api/show and /api/tags, saved to LLAMA_FICHA_MODELO.json:

Property Proven value What is it for?
Configured name llama3:latest Select the local model to write
Learned parameters 8,030,261,248 , approximately 8.03 billion Coefficients of the transformations that process and generate text
Transformer Blocks 32 They apply successive transformations of attention and internal networks
Internal dimension 4.096 Size of the internal representation of each token; is not the 1024 dimensional search vector
Attention/Key-Value Heads 32 / 8 GQA bundled care; multiple heads share keys and values
Width of the internal network of each block 14.336 Intermediate dimension of its nonlinear transformation
Vocabulary 128,256 tokens Units that it recognizes or can issue; It's not 128,256 full words.
Context declared by the model 8,192 tokens Nominal capacity; It does not show that the server reserves that window in each request
Format and quantization GGUF, Q4_0 Tensor packaging and lower precision representation
Installed size 4,661,224,676 bytes, about 4.66 GB Package storage; does not equal total execution RAM
Package footprint 365c0bd3c000a25d28ddbf732fe1c6add414de7275464c4e4d1c3b5fcb5d8ad1 Allows you to identify this version exactly even if you change the alias latest

The same local record reports the legacy option num_keep=24 and three stop flags: <|start_header_id|>, <|end_header_id|>, and <|eot_id|>. They are package configuration; the project did not learn them or change them. The application also does not send num_keep explicitly. For a test that alters the window or template, these inherited options would need to be recorded in addition to those sent in each request.

A learned parameter is a number in a matrix or vector of the model. For example, attention weights transform a representation into queries, keys, and values; internal networks combine features; the output assigns scores to the vocabulary. There is no correspondence "one weight = one thesis": the knowledge learned is distributed. The 8.03 billion Llamas are also not the 4,194,304 parameters of the local encoder.

How an answer is produced

  1. The system retrieves evidence with BM25, vectors or RRF and assigns flags such as [1] and [2].
  2. The code constructs a system message and a JSON with the question and that evidence. For catalog includes metadata; for PDF includes passages and pages.
  3. Ollama applies the model conversation template. The tokenizer converts text into IDs; Llama transforms them into internal representations and processes the context.
  4. Attention relates each position to previous positions. The KV cache preserves keys and values ​​to avoid recalculating the entire prefix at each generation step.
  5. Llama produces scores for the next token; the decoding strategy chooses one, adds it to the sequence, and continues. A token can be part of a word, a sign, or a special unit.
  6. The service receives the text and checks the syntax of the citations. That check does not prove that a claim is supported: the person must check the sources.

In schematic form: question + evidence → tokens → representations → attention and internal networks → next token → cited answer . A fluent output does not imply human understanding, truthfulness or reading of all theses.

Why Llama is used here

The verifiable operational reason is that llama3:latest was already available and working on the server. It allows you to keep the generation local, inspect the configuration and reuse the same API to test other models. It serves as an initial reproducible reference and avoids introducing a dependency on an external API for writing. It was not chosen through a comparison that demonstrated that it is the best model for libraries or for Spanish.

For the user, their contribution is to organize a brief explanation supported by the results. For research, it allows us to separate two questions: "do we recover the correct passages?" and "does the model use them faithfully?" Training another LLM from scratch is not a requirement to study those questions; here their weights are reused and the context is controlled.

Return to index ↑

CHAPTER 13 Llama Settings: what is adjusted and what remains fixed

Execution parameters are options; changing them does not train or modify the weights. This table distinguishes what the code sends from what would be an experimental variable:

Option Status on this project Use and interpretation
model OLLAMA_MODEL=llama3:latest Choose the local package to write
messages A system message and another with a question/evidence Defines instructions and available material
temperature 0 in both collections Favors deterministic selection; does not guarantee accuracy or equality between versions/hardware
num_predict 400 for catalog, 500 for PDF New token limit; may terminate early or truncate response upon reaching it
stream false The application receives the complete wording; showing sources before is not token streaming
keep_alive 15 minutes on this server Temporarily preserve the loaded model
num_ctx Not explicitly sent by the code Effective working window; depends on runtime/configuration and should be registered when experimenting
seed Not explicitly sent Randomization control for reproducible assays
Model top_k Not explicitly sent Limit token candidates during sampling
top_p Not explicitly sent Restrict candidates by cumulative probability mass
repeat_penalty Not explicitly sent Modulates repetition; an excessive value may harm names or citations
stop Inherited from installed package Special marks that end a turn; They do not replace appointment control

Sampling options are described in the Ollama parameter reference; messages, stream and keep_alive are described in the chat API. Current default values must not be attributed to an older runtime version without measuring them.

There are two different top_k: the “Results” field of this website selects references from the search engine; does not change options.top_k of the model. Similarly, temperature=0 decides how to generate text, while temperature 0.1 of contrastive training belonged to the encoder loss.

Reduced example of the structure that prepares the code - the descriptions in square brackets are explanatory markers, not an executed test request:

{
  "model": "llama3:latest",
  "stream": false,
  "keep_alive": "15m",
  "options": {"temperature": 0, "num_predict": 500},
  "messages": [
    {"role": "system", "content": "[Instrucciones: español, sólo evidencia y citas]"},
    {"role": "user", "content": "[JSON con pregunta, fragmentos reales y páginas]"}
  ]
}

Changing OLLAMA_MODEL would select another model already installed. The current form does not expose num_ctx , seed , or sampling; experimenting with them requires a lab client or explicit code modification. This documentation revision does not change those options or download models.

Return to index ↑

CHAPTER 14 Resources and Comparison of Candidate Models

Local inspection found 8 logical CPUs and approximately 31.2 GiB of physical RAM. These resources are shared: Ollama runs on the host, while the search container has its own limits of 4 CPUs and 14 GiB. The container limit must not be interpreted as the RAM available to Llama. The catalog vector index contains about 2.43 GB and also competes for system memory and cache.

Quantization reduces the bits used for many weights. As an indicative calculation, memoria de pesos ≈ parámetros × bits / 8: 8.03 billion at 16 bits would be about 16.06 GB and at 4 bits about 4.02 GB, before scales, metadata, tensors with a different precision and other expenses. Therefore, Q4_0 does not mean that the entire process occupies exactly four bits per parameter. In a previous measurement, the runtime reported about 5.34 GB loaded and 0 bytes of VRAM ; CPU was used. In the metadata inspection of this review there was no model loaded: this reading is not confused with a new performance test.

The necessary RAM/VRAM adds weights, KV cache, buffers and concurrency. A larger context consumes more memory; announcing a long window does not prove that it fits with the rest of the application. The effective window must be consulted with the loaded model, for example using /api/ps, and not deduced only from the sheet. Context in Ollama , loaded models and effective context .

Documentary comparison, not executed benchmark

The following sizes are for tags viewed on September 14, 2026. These are packages on disk , not certified RAM requirements. The candidates are a manageable selection to compare families and sizes; They are not presented as the most recent models nor as a universal ranking.

Model and source Parameters / quantization / package Advantage to investigate Limitation or hypothesis to be tested
Llama 3 8B, current reference 8.03B Q4_0 4.66 GB local Already integrated; allows comparison against a measured starting point It uses CPU and the original tab focuses on English; fidelity needs to be evaluated in Spanish
Llama 3.2 3B 3.21B Q4_K_M 2.0 GB published Minor package and Spanish among its documented languages Hypothesis of lower consumption; It remains to be measured whether it retains appointments and details with fewer parameters
Qwen 2.5 3B 3.09B · Q4_K_M · 1.9 GB published Another multilingual family to contrast instruction tracking and structured data His performance on these theses is unknown; review template and specific conditions of variant 3B
Gemma 3 4B 4.3B · Q4_K_M · 3.3 GB published Another multilingual architecture, with image capacity in addition to text The project sends extracted text, so it does not take advantage of vision; memory and local quality need to be measured
Llama 3.1 8B 8.03B · Q4_K_M · 4.9 GB published Comparison of generation and multilingual support with size close to current No latency reduction is assumed; a larger window requires more resources

Fewer parameters do not guarantee better time or worse quality. Architecture, quantization, tokenizer, generated length, implementation and hardware also influence. Q4_0 and Q4_K_M are not identical formats: a comparison between those packages evaluates the entire configuration, not just the family. The number of tokens is also not directly comparable between tokenizers; Also record response length and solved task.

As a research sequence, first testing Llama 3.2 3B against the benchmark would allow the trade-off between consumption and loyalty to be explored; Qwen and Gemma would expand the comparison between families, and Llama 3.1 would help study a variant of close size. It is a proposal, not a recommendation based on local results from those models. Reusing the API does not guarantee compatibility of all templates or citations without checking them.

Return to index ↑

CHAPTER 15 Llama measurements and execution conditions

The available evidence is from llama3:latest on CPU. Two consecutive short requests, with the same context and the model already loaded, gave:

Actual measurement First request Repetition of the same context
Total time 12,841 s 8,412 s
Reported load 0,084 s 0,074 s
Process context 4,636 s 0,178 s
Generate 47 tokens 8,120 s 8,158 s
Exit speed 5.79 tokens/s 5.76 tokens/s

It is a brief measurement, not a promise for any consultation. The second request can reuse context; A cold initial load was not measured here. Multiple long chunks require processing many more tokens. Existing limits allow up to 400 output tokens for catalog and 500 for PDF; These maximums are not always reached. The metrics separate load, context, and generation. Ollama Metrics .

The previous website canceled the catalog after 65 seconds and the PDF after 90, while the server allowed Ollama 180 seconds. The website now separates recovery and writing: it allows 90 seconds to retrieve and 210 to the generation request, and shows the elapsed time. The server retains its 180 second timeout for Ollama and Nginx allows 210. Showing sources earlier reduces the wait to start reading; it doesn't make the CPU generate tokens faster.

OLLAMA_KEEP_ALIVE=15m allows you to temporarily preserve the loaded model between queries of this server. The code keeps 5m as the default value for other installations. Retaining it reduces reloads, in exchange for taking up RAM for longer; it does not change or train the model. The option is passed to the local API. Ollama keep_alive parameter .

The durations returned by Ollama are expressed in nanoseconds. The muzzle velocity is calculated as eval_count / (eval_duration / 1e9) ; the one in the context uses prompt_eval_count and prompt_eval_duration . The total observed time may also include overhead and waiting. The previous measurement of 24.577 s from the browser corresponded to another catalog question with a reference; is not mixed with these two runs as if they were the same run. Interpretation of metrics.

There are no local times measured for the four candidates in the table. An estimate such as "a 3B model will take half as long" is not invented: that is precisely what the experiment should verify.

Return to index ↑

CHAPTER 16 Proposed experiments: quality, resources and times

Status: proposed protocol; not executed in this update. The two previous Llama measurements were executed; inter-model comparison, human evaluation and subsequent load tests are pending. No models were downloaded, Llama was not trained, and the chat model was not changed.

Questions and hypotheses

Experiment What is changed What is preserved Hypotheses to be tested
E0 · Sources without LLM Disable drafting Question, recovery and evidence The user can resolve certain queries only with sources, with less waiting
E1 · Model Reference and four candidates Ordered Evidence, Instructions, and Short Answer Objective A smaller model could reduce resources while maintaining sufficient fidelity
E2 · Amount of context 1, 3 and 5 references/passages from a fixed list Model, query and generation configuration More evidence could improve coverage, but increase preprocessing and distractions
E3 · Decoding One option at a time: output limit or temperature Model, question and context The length allowed affects cost; more variation does not mean better support
E4 · Charging status First load, hot model with new context and repetition of the same context Hardware and package version Separate loading cost, context processing and prefix reuse
E5 · Concurrency 1 and 2 simultaneous requests Model, questions and limits Shared capacity can affect latency, queues and errors

Tests that download packages, remove the RAM model, or generate load would be prepared in a laboratory environment separate from the utility. Changing the LLM family in E1 does not require rebuilding the indexes: the IDs, order and text of the evidence are frozen to isolate the wording. In E2 the amount of evidence does deliberately change.

Reproducible protocol

  1. Prepare 30 real questions: 12 answerable through catalogue, 12 through PDF passages and 6 where evidence is missing. A librarian defines what data or passages support each answer and when the model should refrain.
  2. Reserve 20 for exploration and 10 for final evaluation, with distribution 8/8/4 and 4/4/2 for those groups. Save the list and a partition seed. Do not choose the model based on the result of the final assembly.
  3. Record commit, index/evidence hashes, model digest, quantization, template, Ollama version, CPU/GPU, RAM/VRAM and effective options. For each condition, keep the rest of the system stable.
  4. Use equivalent instructions in Spanish and the same content of evidence. Set a window that accommodates question, instructions, evidence and output in all models; measure tokens with each tokenizer and record any truncation. In the first comparison you could try num_ctx=4096, only after checking that all the examples fit.
  5. Do three repetitions per question and condition, alternating the order of the models. Separate initial load, new context and repeated context. Also keep errors and truncated responses, not just quick runs that went well.
  6. Measure time to sources, total response time, loading, preprocessing, generation, input/output tokens, and maximum memory. To time the first token a client with stream=true would be needed; the current interface does not measure it.
  7. Submit responses without identifying the model to two reviewers when possible. Evaluate claim support, citation correspondence, coverage, clarity, and abstention; resolve disagreements and preserve denominators.
  8. Publish median and 95th percentile by condition along with spread and faults. With few cases the 95th percentile is unstable; do not extrapolate to the entire user population. Compare quality and time for the same question, not just averages of different texts.

Examples to construct questions: the real record sectesis:000000001 allows asking for author and year; the passage from 0609550.pdf, page 33, allows us to ask what activities it mentions. Asking for results or conclusions absent from these materials serves to evaluate abstention. Expected answers should be reviewed against the complete source available before using them as a reference.

How to interpret a result

A properly formatted citation is not enough. A useful measure is “supported claims/verifiable claims,” accompanied by coverage of expected points and correct abstentions. Also record the proportion of quotes that lead to the passage that supports the claim. A speed improvement would be acceptable only if it meets quality criteria defined before the test; Thresholds should be agreed upon with those who will use the system, not invented after seeing the results.

The experiment deliverable would be one table per model with time, memory, backup, coverage and errors, plus reviewed examples and reproducible configuration. Today the candidates' boxes would be marked "pending measurement" . The advantage of this project for researching applied AI is that it already separates sources, retrieval, representation and generation, allowing each part to be studied without attributing the work of the index to Llama.

Return to index ↑

CHAPTER 17 API contract and request traversal

A API is the interface that allows one program to request an operation from another using defined messages. In this project, the browser sends JSON to FastAPI; The server validates its fields, queries the chosen collection and returns another JSON with results and origin. This separation allows testing the recovery without opening the page and studying its parameters in a reproducible way.

Supported input and default values

Field or control Implemented value and range Where it applies
message Text from 2 to 2,000 characters; extremes are trimmed and unsupported controls are rejected Both collections
method bm25 , dense or hybrid ; API uses bm25 if omitted The website initially selects hybrid , so the initial value of the form differs from the default value of the API
top_k Integer, 1–50; default 5 Number of results; it is not the top_k of the Llama sampling
generate Boolean; default false Llama is only requested when activated and if there are results
year_min, year_max Optional, integers 1500–9999; inverted range is rejected Catalog only
faculty Optional, 1–300 characters; normalized exact match Catalog only
doc_id Optional, pdf- followed by 64 hexadecimal characters PDF only; limit the search to that document
Additional fields extra='forbid' They are rejected; They are not interpreted as new instructions for the server
HTTP Body Up to 32,768 bytes, including bodies sent in pieces /api/v1/sectesis routes middleware; responds 413 if exceeded

The Query and BodyLimit and PDFQuery models implement these controls. The initial value of the web is at index.html . Literal fragment of the catalog contract:

class Query(BaseModel):
    model_config=ConfigDict(extra='forbid')
    message:str=Field(min_length=2,max_length=2000)
    method:Literal['bm25','dense','hybrid']='bm25'
    top_k:int=Field(default=5,ge=1,le=50,strict=True)

Example of a reproducible request to the local installation without a key, not a made-up answer nor a new test executed when writing this example :

curl --fail http://127.0.0.1:8911/api/v1/sectesis/search \
  -H 'Content-Type: application/json' \
  -d '{"message":"problema habitacional México","method":"hybrid","top_k":5,"generate":false}'

For PDF, /api/v1/sectesis/pdf/search is used, with the same core of fields and, optionally, doc_id . The separation of endpoints preserves the distinction between a bibliographic record and a passage of text. There is no simultaneous query of the two collections in that contract.

From sending to visible sources

  1. Validation and access. FastAPI validates the JSON. If the installation configures a key, authorize checks its Bearer; with an empty key, as in this public server, allows querying without it. This access mechanism is not linguistic tokenization.
  2. Available capacity. The catalog supports up to two simultaneous requests on its semaphore; PDF has another two traffic light. Control covers recovery and generation. If busy, reply 429 with Retry-After: 2 ; does not create a persistent queue. They are limits of the software, not a demonstration of how many users it will support without degrading.
  3. Recovery. The service executes the method and filters. The dense index loads on demand, checks corpus and vector correspondence, and keeps the object in memory. That is why the first search may include work that does not appear in the following ones.
  4. Extractive response. Returns references or passages even if no generation has been requested. Each source has an ID and citation number; PDF adds page and endpoint of the original. The JSON identifies answer_mode , dataset_sha256 , retrieved_count , source_status , trace_id , and latency_ms .
  5. Optional wording. If requested, the bounded context of the following subsection is prepared. A build failure returns warning and preserves the extractive response.

Implementation: authorize and catalog paths , Service.search and Service.answer and PDF search path . The UUID trace_id identifies a request; does not demonstrate conversation storage. latency_ms includes the server operations for that request, including generation where appropriate; It is not exclusively the computing time of the model.

The website applies an additional sequence: first sends generate:false, displays sources and only then sends generate:true if requested. Verbatim excerpt from SectesisRequests.search :

const evidence = await post(endpoint, {...body, generate:false}, headers, retrievalTimeout);
onResult(evidence);
if (!body.generate || !evidence.citations.length) return evidence;

The second request fetches back to the server: it does not receive the first results using a cache handle. The behavior is consistent with immutable artifacts, but the program does not automatically compare the identity of the two lists. The browser waits up to 90 seconds in recovery and 210 in generation, displays the elapsed time every second, and retains the first evidence if the second request fails. It does not receive tokens from the LLM one by one: the body sent to Ollama uses stream:false .

Evidence actually sent to Llama

The application does not send the entire database. Select and limit fields before preparing the JSON:

Collection Maximum sources submitted Fields and clippings
Catalog 10 Title up to 1,500 characters, up to 5 authors of 150 characters, year and up to 5 subjects of 200 characters
PDF 8 Citation, file, page and passage number up to 2,000 characters; active fragments have a preparation limit of 1,600

Therefore, asking for 50 results does not cause Llama to read 50 references. The complete PDF, faculty, degree or lexical_text are not sent in the catalog in that context. In PDF, images or the binary file are not sent. These cuts control length by character and by number of sources; does not calculate the model's exact token budget or alone guarantee that any context will fit.

Literal excerpt from generate from the catalog:

context = [{'citation':i, 'title':(d.get('title') or '')[:1500],
            'authors':[str(a)[:150] for a in (d.get('authors') or [])[:5]],'year':d.get('year'),
            'subjects':[str(s)[:200] for s in (d.get('subjects') or [])[:5]]}
           for i,d in enumerate(evidence[:10],1)]

Verbatim excerpt from generate from PDF :

context = [{"citation": c["citation"], "file": c["file"], "page": c["page"], "passage": c["excerpt"][:2000]}
           for c in citations[:8]]

The PDF request incorporates those sources and execution options using this verbatim excerpt from the same function:

body = {"model": model, "keep_alive": os.getenv("OLLAMA_KEEP_ALIVE", "5m"), "stream": False, "options": {"temperature": 0, "num_predict": 500},
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": json.dumps({"question": question, "evidence": context}, ensure_ascii=False)}]}

A system message and a message containing {question, evidence} are sent. No conversation history is sent, even though the interface resembles a chat. keep_alive temporarily keeps the model loaded; it does not add conversational memory to the application. The system message requests the selected language (English or Spanish), evidence and citations, and states that documents are data, not instructions. This is a design measure, not a formal guarantee against incorrect answers or malicious instructions inside a source.

What checks the answer and what a person should check

The functions limit the HTTP response to 2,000,000 cumulative bytes and the text to 20,000 characters. They require at least one citation mark and that the numbers correspond to the sources sent. Literal fragment of validation in catalog :

citations = [int(v) for v in re.findall(r'\[(\d+)\]',text)]
if not citations or any(c<1 or c>len(context) for c in citations):
    return {'ok':False,'error':'Citas ausentes o fuera de la evidencia'}
return {'ok':True,'text':text,'model':model,'citation_check':'syntax_only'}

PDF checks membership of its citation numbers with the same conceptual limitation. This does not verify that each statement is supported, that all are cited, that sources are not contradicted, or that the question is answered correctly. The llm_generated_unreviewed mode identifies it as text without human review. A syntactically valid quote may accompany an erroneous statement; Scientific verification requires contrasting content and source.

When you open the PDF, path files looks for an indexed document, checks that the path is within the configured directory, rejects symbolic links, and compares SHA-256 of the file to the index. A changed PDF returns 409 to avoid presenting another version as the indexed source. The hash tests byte correspondence, not extraction quality or veracity of the content.

Return to index ↑

CHAPTER 18 Server Deployment: Installation, Configuration, and Delivery

The project runs on https://209.126.127.144/demo/sectesis . The code is in VMOsornio/IA , branch main , inside /opt/sites/ia-asistente-virtual/ia-unam . The application runs in a container; the indices and weights are pre-prepared and mounted as persistent files. Query does not retrain or reindex.

How it was implemented on this server

  1. Code and environment. We worked from the branch that incorporated PDF and then integrated the validated version into main. Docker and the existing local Ollama were reused. Dockerfile.final builds a Python 3.11 environment with the project's fixed dependencies; WITH_NEURAL=1 includes local encoder support.
  2. Data preparation. The entire catalog of 593,067 records was preserved. 12,000 were selected for training, the titles were filtered and divided, and 8,750 updated the weights during four epochs. The third was chosen by validation and the entire catalog was vectorized with those fixed weights.
  3. PDF preparation. 50 originals reviewed; The pre-extraction produced 6,541 fragments of 49 files. The final build reused those 49 supported extracts, rebuilt the SQLite BM25, and computed vectors with the newly trained title encoder. Coverage is partial in 32 PDFs. FAQS1.pdf did not produce extractable text. The PDFs and LLM were not used to train another model.
  4. Persistence and containers. The prepared delivery was artifacts/entrega-servidor-entrenada-v1/payload. The service mounts its files as read-only. The hit counter uses another SQLite base on a writable volume, without modifying the bibliographic indexes.
  5. Service publication. Docker Compose runs the browser and a private relay to Ollama. Nginx receives HTTPS and forwards to the application in loopback. Keyless public access was authorized by the owner.
  6. Checking and transfer. Both collections were tested with BM25, dense and hybrid, opening an original PDF and the actual Ollama generation separately. The kit packaged indexes and weights into a private Release, verified the downloaded files with SHA-256, and checked a restore to a new directory.

What each component does

Component Function in actual installation Where it connects or persists
Nginx and HTTPS Receive the browser and publish the web/API public IP 209.126.127.144, port 443; automatic certificate renewal
Service sectesis Serve the interface, find and prepare the generation context 127.0.0.1:8911 on the host → port 8911 of the container
SQLite and vector indexes Recover ready-made logs and fragments Read-only mounts /data , /pdf_index , and /pdf_vectors
Original PDFs Allow opening the source and its page /pdf_raw, read only
private relay Connect the container with Ollama on the host 172.17.0.1:11435 → 127.0.0.1:11434, addresses of this server
Ollama and Llama Generate optional text from evidence Existing local service; llama3:latest
Visit counter Preserve visits and recent presence Volume ia-unam-servidor_visitors → /state/visitors.sqlite

Traversal of a query: browser → HTTPS/Nginx → API → selected index → ​​visible sources → optional Ollama → response . The complete PDF remains available to open, but only the recovered and limited passages are sent to Llama.

The application uses an unprivileged user, read-only container root, and limited permissions. Local configuration limits the browser to 4 CPUs and 14 GiB; Ollama is another host process. The certificates, the particular server configuration and the operational logs remain outside the repository and the Release. The Docker IP and relay must be adapted if installed on another machine.

Execution parameters and resource separation

Configuration Repository common value Applied value or precision of this installation
Python/neural python:3.11-slim, WITH_NEURAL=1 The build report recorded Python 3.11.16 and PyTorch 2.10.0+cpu
Calculation threads OMP, OpenBLAS and MKL: 2 They are variables of the container libraries, not the limit of all host processes
User UID/GID 10001 The application does not run as root inside the container
Finder Network 127.0.0.1:8911 on the host Nginx publishes HTTPS access; application port not binding to all interfaces
Search Engine Resources No CPU/RAM limits on compose.final.yml Local override: 4 CPUs and 14 GiB; Ollama runs apart in the host
File system Read-only root and corpus; Temporary /tmp /state writable exclusively for the counter
Processes / capabilities pids_limit:256, cap_drop:ALL, no-new-privileges:true Parent container limits
health check Every 30 s; timeout 5s; 3 retries Call /health ; checks process life, does not execute all methods
OLLAMA_TIMEOUT_SECONDS 45 s 180s in .env.servidor ; HTTPX option for connect/read/write operations, not global term of all web interaction
OLLAMA_KEEP_ALIVE 5m 15m on this host; retains the model without training it
Ollama Relay Not included in common Compose Local: wait for connection up to 5 s and transfer up to 180 s; private bridge to loopback
Access Optional key, empty by default Public without password by authorization of the owner

Versioned sources: Dockerfile.final , Compose: variables, mounts and limits and build HTTP client . The particular values ​​come from local files compose.servidor.local.yml , .env.servidor – checked non-secret variables only – and operacion-local/ollama_relay.py .

The distinction between liveness and readiness avoids the wrong conclusion: health returns status:'alive' and production_ready:false . The catalog status checks the manifest and shows dense_loaded ; the PDF one checks integrity and counts. A ready:true does not certify relevance, functioning of the LLM, or complete institutional preparation. It is necessary to execute the corresponding checks for each layer.

The artifact report also logs NumPy 2.3.5, scikit-learn 1.8.0, PyMuPDF 1.26.7, FastAPI 0.128.2, and HTTPX 0.28.1. They identify the execution that generated the delivery. Reproducing the same methodology does not imply that any future reconstruction will have identical bytes: commits, versions, hashes, and image must also be recorded; a tag like python:3.11-slim can change over time.

Visits, recent presence and persistence

The counter is a distinct operational component of the AI. The browser creates a random identifier at localStorage and another per page opening. Sends presence every 30 seconds while visible; The server counts identities seen in the last 90 seconds as “online.” A recharge may add another visit. Multiple tabs with the same storage share identity; another browser or device may count separately.

VisitorCounter.record uses SQLite with a BEGIN IMMEDIATE transaction, hashed IDs, a cumulative total, temporary presence and deduplication of page openings over 86,400 seconds. The visitors.js client handles storage, visibility and timing. Literal excerpt of the presence calculation:

ONLINE_SECONDS = 90
VIEW_RETENTION_SECONDS = 86400

Therefore, "people online" is an estimate per browser and time window , not verified people or unique users. These tables do not store queries or IPs; nor do they eliminate bots or automations by themselves. Browser tests count as opens. The ia-unam-servidor_visitors volume retains the total between reboots; It is not mixed with bibliographic SQLite or with weights or vectors.

Paths and variables of this installation

In the table, relative paths start from /opt/sites/ia-asistente-virtual/ia-unam . The local file .env.servidor contains the absolute paths and has permissions 0600.

Variable Source path on this server Route within the application
SECTESIS_ARTIFACTS artifacts/entrega-servidor-entrenada-v1/payload/sectesis /data
SECTESIS_PDF_ARTIFACTS artifacts/entrega-servidor-entrenada-v1/payload/pdf /pdf_index
SECTESIS_PDF_INPUT_DIR data/PDF /pdf_raw
SECTESIS_PDF_VECTORS artifacts/entrega-servidor-entrenada-v1/payload/pdf-vectors /pdf_vectors
SECTESIS_PDF_DENSE_DIR Setting value /pdf_vectors Indicates where to read PDF vectors
SECTESIS_VISITORS_DB Docker Managed Persistent Volume /state/visitors.sqlite

compose.final.yml provides the common configuration. This server adds compose.servidor.local.yml for the published image, limits, and relay. operacion-local/compose.sh sets the name ia-unam-servidor , loads .env.servidor and both Compose files. The files under operacion-local/ are from the operation of this host and are not obtained when cloning.

Build values ​​used here: OLLAMA_HOST=http://host.docker.internal:11435 , OLLAMA_MODEL=llama3:latest , OLLAMA_TIMEOUT_SECONDS=180 and OLLAMA_KEEP_ALIVE=15m . The first address only works because the relay is configured. SECTESIS_API_TOKEN= empty corresponds to the public access of this facility; It is not a key that should be shared.

To check and operate this server already installed , from your directory:

cd /opt/sites/ia-asistente-virtual/ia-unam
./operacion-local/compose.sh config --quiet
./operacion-local/compose.sh ps
./operacion-local/compose.sh logs --tail 50 sectesis

When you publish a previously built and validated image, its local reference is updated and applied with ./operacion-local/compose.sh up -d --no-build --pull never --wait sectesis . That operation recreates the service when the image changes; does not rebuild indexes. A git pull alone updates checkout files, but does not replace the already running image.

How the kit turned artifacts into a verifiable delivery

The kit is an automation of the modules explained in this guide. It does not replace training or indexes with a different technique: it invokes its functions and commands prepare, check, package, publish and restore.

Stage and function Action implemented Evidence or limit
source_identity Checks Git root and absence of pending tracked changes; records commits and hashes from selected sources It is not an inventory of all files on the host; SHA list includes Python, requirements, Compose and Dockerfile
build It demands a new destiny; reuse catalog/weights or prepare TXT and train if requested; index and vectorize PDF --retrain is explicit; In this installment, titles were trained. Does not modify LLM weights
functional_check ASGI TestClient against real application, two queries derived from titles/passages per collection, three methods, top_k:3 , generate:false Check operation on real data; It is not a human benchmark or Ollama execution
package Tar.gz, compression level 3; parts of 128 MiB; manifest sizes and SHA-256 It does not claim that a hash certifies content quality; preserves provenance of packed bytes
upload Requires confirmation and private repo; check remote commit; create draft, download and compare assets; publish with --publish Register real URL and download_sha256_verified ; large data out of Git history
restore Validates parts, names and sizes; extracts temporarily; check internal hashes and switch to new destination Does not install Docker/Ollama, does not train and does not overwrite a previous release

Functional testing derives questions from useful early texts, asks for three outcomes, and demands some evidence in each combination of collection and method. One of the checks opens an original PDF and checks the %PDF header. It is used to detect broken or incompatible indexes; it does not demonstrate that the search engine finds what people need in an independent set of questions. Verbatim extract from the request in functional_check:

responses = [client.post(endpoint, headers=h,
    json={"message": q, "method": method, "top_k": 3, "generate": False}) for q in qs]

Report files have different times and meanings:

Record What documents in this delivery
payload/BUILD_REPORT.json Counts, training, PDF, ASGI versions and tests; llm_inference:false refers to that phase
BUILD_STATUS.json prepared_verified_not_uploaded : status at the end of build , before upload; the uploader does not rewrite it
UPLOAD_RESULT.json Release published ( isDraft:false ), tag, commit, URL and download verification; credits the subsequent phase
operacion-local/VERIFICACION_HTTPS_ENTREGA.json Real independent verification that Ollama did execute on both collections
operacion-local/VERIFICACION_HTTPS_GUIA_COMPLETA.json Post check of public page/API, six lookups and PDF hashing; That run did not generate again with Ollama
operacion-local/VERIFICACION_GUIA_COMPLETA_PUBLICA.json Viewable content and mobile/desktop navigation; not a human evaluation of relevance

That BUILD_REPORT says "No Ollama inference" and a subsequent report proves generation is not a contradiction: they describe different phases. Unit tests can use mock fixtures and transports to test logic; They are not presented as the delivery PDFs. Local evidence is not copied indiscriminately to Git because it may contain operational information; The manual summarizes their scope and maintains their names for the owner's audit.

The data artifact comes from commit 9193ce88f586783d3d8ab5dd93ec6880f49116a9 ; The code revision cited in this manual is 01bbc0ccb4d79ff5b829420e0d67f032827c099b. Subsequent documentation/interface improvements do not amount to new training. This separation allows the application to be updated and still use a data delivery identified by hashes.

How to install a copy on another computer

Requires Git, authenticated GitHub CLI with access to the private repository, Python 3.11 or later to restore, and Docker with Compose to run. The following commands are used in a new directory on the target computer; They are not cloned again on top of the existing installation. Build can download software dependencies, but restore does not download models or train.

gh auth status
gh repo clone VMOsornio/IA IA-unam -- --branch main --single-branch
cd IA-unam
gh release download ia-unam-artifacts-20260914-0428 --repo VMOsornio/IA --dir artifacts/descarga-artefactos
python3 artifacts/descarga-artefactos/ia_artefactos.py restore --assets artifacts/descarga-artefactos --out artifacts/restaurados-v1

The download and restore destinations must be new. The package contains 20 parts, 2,660,066,142 bytes compressed and 4,760,857,985 bytes restored . The restorer checks SHA-256 of the shares and files. Before extracting, require free space for the expanded size plus 15% margin and a temporary copy of the compressed file; The existing download also takes up space. The Release preserves the commit with which its data was prepared; main contains the supported updated interface. Does not include the raw TXT or the Ollama model. The 50 original PDFs already versioned under data/PDF are obtained with the clone; no new originals were added to the Release. That assembly must be preserved to open the sources, in addition to restoring their derived data. The payload has 19 files; The 20 parts are segments of the compressed file, not 20 models or 20 different datasets.

Create a new .env.servidor file by replacing /ruta/IA-unam with the actual absolute path of that copy:

SECTESIS_ARTIFACTS=/ruta/IA-unam/artifacts/restaurados-v1/sectesis
SECTESIS_PDF_ARTIFACTS=/ruta/IA-unam/artifacts/restaurados-v1/pdf
SECTESIS_PDF_INPUT_DIR=/ruta/IA-unam/data/PDF
SECTESIS_PDF_VECTORS=/ruta/IA-unam/artifacts/restaurados-v1/pdf-vectors
SECTESIS_PDF_DENSE_DIR=/pdf_vectors
SECTESIS_PORT=8911
WITH_NEURAL=1
SECTESIS_API_TOKEN=
OLLAMA_MODEL=
OLLAMA_TIMEOUT_SECONDS=180
OLLAMA_KEEP_ALIVE=15m

With OLLAMA_MODEL empty you can try recovery without drafting. To add generation, configure a local Ollama with a model already available and an address accessible from the container. On Linux, a service bound only to 127.0.0.1 on the host is not automatically made reachable by host.docker.internal; An appropriate private connection is required, such as the relay in the installation described. The capabilities of the model and its templates should be checked before using another package.

chmod 600 .env.servidor
docker compose -p ia-unam --env-file .env.servidor -f compose.final.yml config --quiet
docker compose -p ia-unam --env-file .env.servidor -f compose.final.yml build sectesis
docker compose -p ia-unam --env-file .env.servidor -f compose.final.yml up -d --wait sectesis

The copy is available locally at http://127.0.0.1:8911/demo/sectesis. The empty value of SECTESIS_API_TOKEN leaves queries keyless on that local access; a private installation can establish its own key. Publishing to the Internet requires configuring HTTPS proxy and new server access; clone does not install your certificates or automatically copy the origin server infrastructure.

Initial checks for that keyless local configuration:

curl --fail http://127.0.0.1:8911/health
curl --fail http://127.0.0.1:8911/api/v1/sectesis/status
curl --fail http://127.0.0.1:8911/api/v1/sectesis/pdf/status
curl --fail http://127.0.0.1:8911/api/v1/sectesis/visitors

Then, query both collections using all three methods, review snippets, and open a PDF. /health only checks that the process is responding, not the relevance of the responses or the complete state of the data. For a keyed installation, API routes require appropriate authorization. The counters are persisted in the Compose project volume; Do not delete it with down -v if you want to keep the visits. Do not add databases, indexes, weights, or private configuration files to Git history.

Return to index ↑

CHAPTER 19 Contributions, benefits, limitations and learning from the laboratory

The result is a functional AI laboratory applied to library data and thesis , with a public interface and a reproducible process. It allows you to observe how data is converted into indexes, how an encoder learns a representation, and how an LLM uses retrieved evidence to write.

Available contributions and usefulness

Contribution available Benefit Limitation
Normalized catalog of 593,067 records Search files by topic, author and filters, reusing a common structure Metadata may be incomplete; a file does not contain the full text of the thesis
6,541 fragments of 49 PDFs with text Reach specific passages and open the document to the corresponding page No new OCR was applied; tables, images, reading order and documents without text require review
Search BM25 Find lexical matches on metadata or passages It depends on the terms used and may lose relationships expressed with other words
Local encoder and dense search Study a learned representation, inspect weights and compare similarities Trained with titles, limited vocabulary, maximum of 64 useful tokens and presence of zero vectors
RRF Hybrid Search Combine BM25 and dense positions in the chosen collection Combining methods does not guarantee better results on all queries
Local writing with Llama and visible sources Organize an initial orientation and allow evidence to be reviewed while writing You may omit, misinterpret, or invent; a syntactically valid citation does not ensure support
IDs, pages, hashes and manifests Relate each result to its source data and check versions The integrity of the file does not certify veracity, relevance or quality of extraction
API, Containers and Private Release Run and move code, indexes and weights separately The resources and infrastructure of the destination computer determine the operation
Technical guide, examples and glossary Explain and reproduce laboratory decisions with real materials It is prototype documentation; It does not replace training, user evaluation or specialized review
Persistent counters Observe page opens and recent presence These are browser estimates, not identified people or audited statistics.

What was developed and what was achieved

Bibliographic parser, JSONL normalization, SQLite FTS5 indexes, encoder training and inference, PDF extraction and fragmentation, lexical/dense/hybrid retrieval, API and interface with sources were developed. Integrated local generation using Ollama, syntactic validation of citations, publication and restoration of artifacts, containers and persistent counters.

The delivery managed to make the complete catalog of this TXT and the fragments extracted from the available PDFs searchable. The encoder has 1,024 dimensions and was trained for four epochs with the indicated title partition; all catalog titles and PDF fragments were then rendered with fixed weights. The service now allows you to consult, compare methods and open sources. API, browser, integrity, persistence, and generation checks confirm this; are not a human measurement of accuracy or institutional usefulness .

What was learned and what knowledge was practiced

Documented technical learning is expressed in decisions and results that can be reviewed in the code:

Worked knowledge Concrete application in the laboratory
Prepare data before applying AI Interpret ALEPH, normalize fields and preserve provenance and missing items
Distinguish document, record, page and fragment Do not present 593,067 files as read theses or 6,541 fragments as different documents
Build information retrieval Use inverted indexes, BM25, filters and sort criteria
Understand training Separate sample, partitions, loss, gradients, epochs and selection by validation
Distinguish weights from representations Train a transformation and then calculate vectors for the entire corpus without further tuning
Understand transfer and its limits Apply an encoder trained with titles to PDF passages and recognize vocabulary and tokens it does not cover
Integrate RAG Send bounded evidence to the LLM, maintain sources and distinguish retrieval from writing
Evaluate carefully Separate functional tests, performance measurements and human judgments of relevance
Operate a reproducible application Separate code, configuration, persistent data and container images; verify hashes when moving
Formulate comparable experiments Set questions, evidence and conditions before comparing models, quality, times and memory

These insights can be reviewed in code, configuration, and examples. The laboratory allows you to go through the complete process and detect where a result originates: extraction, segmentation, index, representation, recovery or generation.

Scope of the project and value of the laboratory

The current scope includes record and passage search, retrieval comparison, optional writing with sources, and reproducible deployment . It does not cover the entire UNAM collection, all library management processes, visual reading of pages, general OCR, training for one's own LLM, or answers whose accuracy has been guaranteed. Nor does it automatically link each catalog record with a PDF or consult both collections at the same time.

Its usefulness as a laboratory is to offer a real basis for learning, demonstrating and experimenting: a research team can follow the transformation of the data; a developer can reproduce the service; a user can search and check sources; and a librarian can help evaluate whether the results respond to real needs. It is a basis for investigating improvements and not a statement that AI replaces library judgment.

The core knowledge gained in the process is that applying AI requires organizing the data, defining what is retrieved, controlling what the model receives, and checking the result . A learned representation or a well-written response only adds value when its origin and limits are understood. The experiments in the previous section allow us to study this value without confusing technical performance with demonstrated quality.

Query code: search.py , dense.py , service.py , app/pdf_rag/engine.py and both generation.py . Local evidence: BUILD_REPORT.json , HTTPS and browser verification logs, OLLAMA_DIAGNOSTICO_TIEMPOS.json , EJEMPLOS_PROCESO_REAL.json , EJEMPLO_FRAGMENTOS_CONSECUTIVOS.json , and ejemplo-encoder-real.json . See the full glossary .

Return to index ↑

CHAPTER 20 Pending research: what to change, why and how to check it

This section is a research program, not a description of work already performed. The values ​​used constitute a reproducible laboratory configuration. No controlled comparison was found that demonstrates that 1,600 characters, 1,024 dimensions, contrastive temperature 0.1 or four epochs are optimal for these theses. Explaining the function of a parameter justifies its technical role; Demonstrating that its value is the best requires trials.

Existing options that do not yet have results in this delivery

The repository includes index_pretrained - allows creating a catalog index with an already locally available SentenceTransformer , identifying model, revision and files. It uses CPU, local_files_only=True , trust_remote_code=False , encoding batches of 8 and a maximum sequence of up to 512 tokens, also limited by the model itself. You can use semantic_text and a query prefix. That route was not activated here. It is also not a direct replacement for the entire system: the PDF vectorization and the kit in this delivery expect the teaching encoder. Applying it to PDF or packaging it requires adapting and testing those routes. Code: index_pretrained , lines 78–110 , PDF model check .

You can also change the query method, top_k , filters or enable/disable untrained wording. Changing the text used by the encoder, its vocabulary or its weights requires calculating compatible vectors again. Changing the fragmentation changes the recovery units: it forces rebuilding the PDF index and its vectors in a new version, and revising the relevance judgments whose IDs depended on the previous fragments.

Hypotheses by stage

The alternative values ​​in the table are illustrative candidates for designing a test, not a validated recommendation or commands that have been executed.

Research question Candidate variable or technique What should be developed or prepared How to check its usefulness
Is information recovered from skipped pages? OCR only on pages without text or with faulty extraction Integrate OCR, register engine/language and preserve provenance; create new shards and indexes Compare transcript against reviewed pages and measure relevance; count errors and cost per page
Do current cuts separate necessary evidence? Fragments of 800/1,600/2,400 characters and overlaps 100/200; or cuts per paragraph New versions per condition, without mixing arrays; review how annotations are moved Recall and nDCG of passages, response support and context cost with the same questions
Does the representation retain enough information? Titles versus title + subjects/abstract; a limit of 64 versus another defined limit Modify/version representation, train only with train, generate all matrices again Vocabulary coverage, zero vectors and recovery by topic, without choosing test conditions
Does the trained layer provide value? BM25, TF-IDF cosine without a network, and the contrastive encoder Implement a comparison with the same sources, partitions, filters and queries Measure relevance and cost by method; contrastive loss reduction is not enough
Is another encoder capacity better? 256/512/1,024 dimensions, different learning rates or temperatures Independent trainings with registered seeds and resources; new vectorization Select with validation; evaluate reserved test once the configuration is set; report memory and dispersion
Will another merger help? Vary candidates by method and RRF constant; compare each isolated method Version the configuration and retain results and timings for each query Measure paired improvements and regressions, especially queries solved by only one method
Can search time be reduced for a larger corpus? Approximate ANN index versus exact search Integrate an engine, version its index and check compatibility with filters Recall relative to the exact ranking, human relevance, latency and memory; not just speed
Does a second reading of candidates improve ranking? A reranker that scores the question and passage together Add a stage for the top candidates and a compatible model nDCG/recall and additional time; it cannot recover evidence that never entered its candidate set
Can users navigate from a catalog record to a thesis? Explicit catalog–PDF links using verifiable identifiers Create and review a mapping table with provenance and ambiguous cases Link accuracy and coverage; do not infer equivalence from similar names alone
What does a person gain from using it? Search tasks with and without the assistant Define tasks, consent, librarian review and a comparison protocol Time to a valid source, task success and errors; distinguish website visits from people and usefulness

The proposed OCR can be investigated using Page.get_textpage_ocr, a PyMuPDF capability that requires Tesseract; the current code calls text extraction, not that method. Official OCR reference.

Faiss provides exact and approximate search families with different memory, cost and recall tradeoffs. It is not integrated here; a future integration should be compared against the project's exact dot products and respect filters before returning results. Official Faiss documentation.

A cross-encoder computes a score by processing the query and candidate together. It can be studied as a second stage for a small number of candidates, at additional cost per pair. It was not used in this installation. Official Sentence Transformers documentation.

A common protocol for distinguishing changes from improvements

  1. Formulate a question and hypothesis before running an experiment: for example, whether different chunking retrieves more relevant passages within an agreed time limit.
  2. Fix the data and provenance: commit, hashes, parser version, passages, encoder, dependencies and resources. Separate development and test sets; group questions from the same document where appropriate to prevent leakage between groups.
  3. Prepare reviewed relevance judgments with explicit criteria and recorded disagreements. For PDFs, annotate the document/page and evidence; if chunks change, review their correspondence. Include questions without evidence in a separate protocol: the current evaluator requires at least one positive per case.
  4. Change one variable per initial trial and keep the others comparable. When studying interactions, specify a factorial design; do not attribute an outcome to a single variable when several changed.
  5. Record every run, including errors. For latency, distinguish startup/lazy loading from queries with data already loaded, alternate method order and repeat. Warming up one search can benefit the next.
  6. Compare relevance, cost and failures per question; report set size, variability and regression examples. If intervals are estimated by resampling, document the sampling unit and procedure. This delivery does not include automatic statistical significance calculations.
  7. Choose configurations using development/validation data; consult the test set after they are fixed. Retain the previous system for comparison and publish the conclusion together with its limitations and reproducible files.

A possible scientific result is a bounded answer to a question, backed by verifiable evidence. It may show an improvement, a tie, or that a technique adds cost without benefit; all three outcomes are informative. The Llama experiments described above complement these retrieval and preparation tests without turning them into LLM training.

Return to index ↑

CHAPTER 21Map of files, functions and evidence for studying the code

The following references pin the audited commit 01bbc0c. Function names locate the operation; output files show what it produced. Configuration and manifests determine which path was actually used.

Stage File and entry point Evidence or output to inspect
Parse and normalize TXT ingest.py: records, canonical, prepare sample.jsonl and profile.json; source count and hash
Catalog index and BM25 search.py: build, LexicalIndex.search, rrf catalog.sqlite: records, docs and manifest tables; ordering and filters
Train and encode titles neural.py: Encoder, split_of, train encoder.json, weights.pt, splits.json, ids.json, vectors.npy
Select and vectorize the full catalog ia_artefactos.py: reservoir, train_and_encode Selection of 12,000 records, training fingerprint and final matrix of 593,067 rows
Dense catalog retrieval dense.py: DenseIndex ID/vector correspondence, integrity, filters, dot product and ranking
Extract and chunk PDFs pdf_rag/ingest.py: Options, extract, chunks, index_pdfs pdf/manifest.json, chunks.jsonl and SQLite; statuses, pages, offsets and reuse
Vectorize and search PDFs pdf_rag/engine.py: vectorize, Engine.search pdf-vectors/manifest.json, copied encoder and a matrix of 6,541 rows
Coordinate catalog retrieval and generation service.py: search, answer Method, evidence, answer_mode, warnings and API response latency
Context and citation validation sectesis_ai/generation.py and pdf_rag/generation.py Two messages to the LLM, options, returned text and syntactic checks of [n]
Evaluate catalog ranking evaluation.py: metrics, evaluate Reviewed cases and positives, per-case/aggregate metrics and timings; not human certification
Package, verify, restore and upload ia_artefactos.py: functional_check through upload BUILD_REPORT.json, BUILD_STATUS.json, release manifest and checksums, UPLOAD_RESULT.json
Run and mount persistent storage compose.final.yml and Dockerfile.final Image, mounted paths, allowed variables, container state and visit-counter volume

To repeat a local inspection without printing the entire corpus, from the repository root:

# Leer las funciones auditadas y la configuración común.
sed -n '39,91p' app/sectesis_ai/neural.py
sed -n '49,67p' app/pdf_rag/ingest.py
sed -n '170,205p' app/pdf_rag/engine.py
cat compose.final.yml
# Leer reportes de esta instalación; adaptar la ruta en una copia restaurada.
python3 -m json.tool artifacts/entrega-servidor-entrenada-v1/payload/BUILD_REPORT.json
python3 -m json.tool artifacts/entrega-servidor-entrenada-v1/BUILD_STATUS.json
python3 -m json.tool artifacts/entrega-servidor-entrenada-v1/UPLOAD_RESULT.json

Kit reports describe a delivery; they do not update with every visit or documentation change. The running image, interface commit and data-preparation commit may differ and should be recorded separately. Pinned dependencies are in requirements-final.txt and requirements-final-ml.txt. Numerical reproducibility also depends on platform and versions: a seed alone does not promise bit-for-bit identity on every machine.

Return to index ↑

QUICK REFERENCEGlossary: terms and processes in this project

Definitions tied to this project's code. A text “token” and an “access token” are different concepts. The current public chat does not require an access key.

Term Meaning here Example or connection to the process
Collection The materials held by a library or institution. This TXT file and 50 PDFs are the available inputs; coverage of all UNAM holdings is not claimed.
Bibliographic catalog Organized descriptions of documents. A thesis's title, author, year, degree, faculty and links.
IATESIS NOETILUM applied research project developed by Víctor Manuel Osornio Luna. An AI laboratory for bibliographic records and thesis passages; uses the sectesis.txt source.
SECTESIS Technical source name and identifier namespace used by the repository. sectesis.txt and sectesis:000000001; it is not an algorithm and no official expansion is invented.
Record A unit describing a document in the catalog. One canonical line in sample.jsonl.
Metadata Data describing a document. A publication year is not the text of the thesis.
Dataset Data organized for processing. The complete catalog JSONL or the set of PDF passages.
Corpus A collection of texts or records used by a system. 593,067 records from this TXT file.
Sequential ALEPH Export format recognized by the parser through lines, IDs, tags and subfields. 000000001 24510 L $$a…; this parser does not read binary ISO2709.
Tag and subfield Markers indicating the meaning of bibliographic data. 245 $$a for title; 100 $$a for author.
Parser Code that interprets the file structure. records() groups lines; canonical() assigns fields.
Ingestion Controlled incorporation of data into the process. Read, check the format, normalize and save.
Normalization Applying consistent rules to text. Whitespace and Unicode; search normalizes case and accents.
Canonical record A consistent internal representation of the original record. id, title, authors, year fields and quality flags.
JSON A data format with objects, lists, strings and numbers. An API response.
JSONL One JSON object per line. Allows iteration over a dataset without loading it entirely into memory.
SQLite A database engine storing tables and indexes in a file. catalog.sqlite; it is not a neural network.
SQL A language for querying and organizing the database. SELECT … WHERE … ORDER BY … LIMIT.
FTS5 SQLite extension for full-text search. docs and fts virtual tables.
Inverted index A structure connecting terms to documents and occurrences. Allows retrieval of records containing “habitacional”.
Indexing Building structures to retrieve data. Creating FTS5 and its statistics does not imply training.
BM25 Best Matching 25: ranks lexical matches by frequency, rarity and length. A word-based retriever; it does not draft answers.
Lexical retrieval Search based on terms in the text. BM25 can find a surname present in metadata.
Text token A text unit used by a processor. The encoder uses normalized words; Llama uses its own tokenization, which does not correspond to whole words.
Tokenization Splitting text into units. tokens() discards short terms and some common words; returns up to 64.
Stopwords / excluded words Words omitted by the query and encoder tokenizer. “de”, “la”, “tesis”; FTS5 has its own tokenizer and does not use the same list.
Vocabulary The set of features recognized by the vectorizer. Up to 4,096 terms learned from training titles.
TF A term's frequency in a text. How many times a word appears; the active TF-IDF uses sublinear scaling.
IDF A weight depending on the term's frequency in the dataset. In the vectorizer, it is fitted only on training titles.
TF-IDF A numerical representation combining TF and IDF. 4,096 input features for the linear layer.
Feature An input variable for a model. A position in the TF-IDF vector corresponds to a vocabulary term.
Sparse vector A vector with many zero entries. The example title has nine nonzero TF-IDF features.
Encoder A model transforming an input into a representation. TF-IDF → linear layer → normalized vector; it does not generate prose.
Linear layer Multiplication by a parameter matrix. y = W @ x, without a bias in this encoder.
Weight / parameter A number adjusted within the network during training. weights.pt stores 4,194,304 encoder parameters. BM25 column weights are different: they are configured, not learned here.
Hyperparameter A choice in the training procedure. Four epochs, batch size 96, learning rate 0.002; not a learned weight.
Embedding The resulting vector representing a text. 1,024 numbers per title or passage.
Dimension Number of components in a vector. 1,024 does not mean 1,024 documents or words in an answer.
L2 normalization Adjustment of a vector's Euclidean length. Nonzero vectors are scaled to norm 1.
Zero vector A representation with no useful numerical signal. It cannot support a useful dense comparison; the text may still be available in BM25.
Sample A selected subset of the dataset. The initial 12,000 records selected to prepare training.
Reservoir sampling Uniform sampling while iterating over a file. Avoids loading the entire catalog into RAM to select 12,000 records.
Seed A value that makes random operations reproducible. 20260909 for selection and training.
Deduplication Grouping or detecting repeated entries according to a rule. Identical normalized titles are grouped before splitting the training data.
Training set The partition used to fit the model. 8,750 titles in this delivery.
Validation The partition used to compare training decisions. 1,144 titles; used to select epoch 3.
Test set The partition reserved for separate evaluation. 1,116 titles; not used to select the epoch.
Data leakage Improper use of evaluation information during training or selection. Fitting TF-IDF on all titles before splitting would cause leakage; here it is fitted on the training set only.
Time One pass through the training examples. Four passes; not four complete readings of every PDF.
Batch A group processed in one step. Up to 96 titles.
Self-supervision Constructing learning signals from the data itself. Two views of the same title, without human relevance labels.
Contrastive learning An objective that brings positive representations closer and distinguishes others. Masked views of the same title versus other titles in the batch.
Masking / augmentation A controlled alteration to create a training view. Masking features with probability 0.15.
Loss A measure of error for the training objective. Lower validation loss selects a checkpoint; it is not an accuracy percentage.
Cross-entropy A loss function comparing predictions with targets. Applied in both directions between views in the batch.
Temperature A parameter that changes a score distribution. 0.1 in the contrastive objective; generation temperature 0 belongs to a different process.
Gradient / backpropagation Derivatives indicating how to adjust parameters. loss.backward() computes gradients.
AdamW optimizer A rule updating weights using gradients and regularization. opt.step() during training.
Learning rate The reference step size for the optimizer. 0.002 in this run.
Regularization / weight decay A penalty or constraint helping control fitting. Weight decay of 0.001 with AdamW.
Checkpoint A saved state of the model weights. The weights selected at the end of epoch 3.
Inference Using an already fixed model to produce output. Vectorizing all titles and PDF passages; also generating text with Llama.
Vectorization Computing vectors for inputs. Applying fixed weights to 6,541 passages does not train those weights again.
Cosine similarity A comparison of the directions of two vectors. For unit-norm vectors, it equals their dot product.
Dense retrieval Search by comparing vector representations. Exact comparison with NumPy in this project.
ANN Approximate nearest-neighbor search. Not used by the active backend, which compares in blocks.
Ranking / score An ordered list and the score used for ordering. The score is not a probability of truth or certified relevance.
Top-k Maximum number of requested results. Five references by default in the interface.
Filter A restriction on which records may compete. Year and exact faculty in the catalog; not available in PDF search.
RRF / hybrid search Fusion of ranking positions. Adds 1/(60+posición) from BM25 and dense search within the same collection.
PyMuPDF A library that opens PDFs and extracts their available text. Page-by-page extraction using get_text.
OCR Recognition of characters in images. No additional OCR was run; an image without a text layer may remain unindexed.
Passage / chunk A text unit indexed for retrieval. Up to 1,600 characters from one PDF page.
Overlap Text shared by neighboring chunks. A target of 200 characters, adjusted to word boundaries.
Physical page / page label Position in the file / label that the PDF may declare. Physical page 33 may have a different printed page number.
Offset A character's position in normalized text. char_start=0, char_end=1596, exclusive end.
ID Identifier of a record or passage. sectesis:000000001 or pdf-…:p33:c0.
SHA-256 / hash A fingerprint computed over bytes. Detects changes; does not establish truth, quality or authorship.
Manifest / integrity A file recording contents, versions, counts and fingerprints. Prevents accidental mixing of vectors from different versions.
Artifact A file produced by preparation, training or indexing. SQLite, JSONL, weights, vectors and reports.
float32 / NumPy NPY 32-bit number / array format. Each value occupies four bytes; vectors.npy preserves shape and type.
mmap Reading files through memory mapping. Allows block-by-block access to the vector matrix.
LLM A language model that produces sequences of tokens. Llama 3 drafts using the context it receives.
Transformer An architecture combining attention and internal transformations of representations. Llama has 32 blocks; the local encoder is a linear layer, not a Transformer.
Autoregressive Generates each token conditioned on preceding tokens. The answer is built step by step until completion.
Attention An operation weighting information from other permitted text positions. Connects question tokens with the available context.
GQA Grouped-query attention sharing keys and values. The local Llama metadata records 32 query heads and 8 key/value heads.
KV cache Temporary key/value memory used during generation. Avoids recomputing the entire prefix; it is neither training nor a bibliographic database.
Logits Vocabulary scores before selecting the next token. Not BM25 relevance scores or probabilities of truth.
Decoding The procedure for choosing tokens from model scores. Generation options belong to this stage and do not update weights.
Prefill / context processing Initial computation over input tokens. Measured separately from output-token generation.
Context window / num_ctx Working space for the token sequence. The 8,192 declared by the model does not establish the runtime's effective window.
num_predict Maximum number of new tokens the answer may generate. The code sends 400 for the catalog and 500 for PDFs.
Sampling top_k Number of candidates considered when choosing a token. Different from the search top_k, which selects references.
top_p Restriction of candidates by cumulative probability during sampling. Not explicitly configured by the current clients.
Token streaming Progressive delivery of text as it is generated. The application uses stream=false; showing sources first is a different mechanism.
TTFT Time to the first observed output token. Requires streaming instrumentation; not measured by this interface.
95th percentile / p95 A value below which approximately 95% of measurements fall. Unstable with few runs; report the number of cases and failures alongside it.
Abstention Acknowledging that the evidence cannot support an answer. Preferable to attributing conclusions to a passage that does not contain them.
Llama / Ollama Language model / program that runs it locally. llama3:latest is the model; Ollama exposes /api/chat.
Q4_0 quantization A compact representation of Llama model weights. Reduces memory compared with higher-precision weights; does not describe the encoder's float32 values.
CPU / GPU / VRAM General-purpose processor / accelerator / accelerator memory. Llama uses the CPU and reported 0 bytes of VRAM in this installation.
Prompt / context Instructions and data sent to the LLM. The question and selected references, not the entire catalog.
RAG Retrieval-augmented generation. Retrieving evidence before asking Llama for an answer.
Fine-tuning Adapting a previously trained model with new data. Llama was not fine-tuned; the experimental encoder was trained from scratch.
Hallucination A generated claim without sufficient support. A correctly formatted citation does not guarantee that the passage supports the claim.
Citation / traceability A reference that lets users return to the source. Document, page, passage and ID.
Latency / timeout Response time / waiting limit. Generation has a longer allowance than retrieval; sources appear first.
keep_alive How long the loaded model is temporarily retained. Avoids some reloads but keeps RAM occupied.
API / ASGI HTTP interface / communication protocol for the Python application. FastAPI and TestClient verify requests; ASGI does not establish human-assessed quality.
Access token A key used to authorize requests. Different from text tokens; the current public chat does not require this key.
HTTPS / Nginx Encrypted transport / server accepting public connections and forwarding them. The public IP reaches the backend listening on loopback.
Docker / Compose Containers / a joint definition of services and mounts. Run the app with artifacts mounted read-only.
Git / commit / branch Code history / identifiable revision / line of work. UI code can evolve without retraining.
Release / tag A downloadable delivery / reference to a code revision. Large weights are assets in the private release, not stored in commits.
Fixture Controlled data used to test code. Not used as real theses in this delivery.
Precision@k / Recall@k Fraction of relevant results / fraction of relevant items retrieved. Require defining which results are relevant to each question.
MRR / nDCG Metrics for the first relevant result's rank / normalized ranking quality. The evaluator uses binary relevance, not graded relevance; requires positives for each query.
Human evaluation People reviewing usefulness, relevance and support. Questions and relevance judgments from librarians are still needed.
min_df Minimum number of documents containing a feature for it to be retained. Set to 2 training titles when fitting TF-IDF; not the word's total frequency.
max_features The vectorizer's vocabulary limit. Set to 4,096; limits input to the linear layer.
sublinear_tf Transformation of positive frequency to 1 + ln(frecuencia). Enabled in TF-IDF; reduces growth caused by repetition.
smooth_idf Smoothing the IDF calculation to avoid problematic divisions. Uses ln((1+n)/(1+df)) + 1 with training documents.
AdamW betas Coefficients for moving averages of the gradient and its square. The optimizer retains defaults of 0.9 and 0.999.
Epsilon / eps A small constant for numerical stability; depends on the operation. AdamW uses 1e-8; PyTorch normalization uses 1e-12; neither is a relevance threshold.
Exact search Computes comparisons across all eligible candidates. The dense search engine scans blocks of 4,096 vectors; does not use ANN.
Reranker / cross-encoder A second stage that reranks candidates; a cross-encoder processes the query and candidate together. A future proposal; no model or stage of this kind has been run here.
Ablation A comparison removing or replacing a component to study its contribution. Comparing TF-IDF without a network against the encoder remains a pending experiment on this corpus.
Benchmark / proxy evaluation A set of comparison tasks / a simplified substitute for real tasks. Queries derived from titles do not replace user judgments; the historical proxy belongs to a different corpus.
Cold start / warm-up First load / advance resource preparation before measurement. The first dense query includes verification; subsequent measurements may be faster.
Concurrency semaphore A mechanism limiting simultaneous operations. Two slots per collection and process; a busy server returns HTTP 429, not a per-person quota.
Liveness / readiness A running process / components ready for an operation. /health and index statuses have different scopes; dense retrieval is checked when loaded.
Experimental hypothesis A bounded statement an experiment can support or refute. For example, different chunking might improve retrieval; it would still need measurement.
Paired comparison Compares methods using the same questions or units. Shows per-case improvements and regressions without mixing different tasks.
Factorial design An experiment combining levels of several variables to study effects and interactions. A methodological proposal; no factorial hyperparameter search was performed in this delivery.

The process guide shows how these terms connect with the actual data and output files. Implementation definitions are grounded in the project code; for FTS5 and Ollama metrics, see the references linked in that guide.