Sioyek is a PDF viewer with a focus on textbooks and research papers - ahrm/sioyek

The more interesting part for me (esp as a computer vision at heart who is temporarily masquerading as a natural language person) is whether pixels are better inputs to LLMs than text. Whether text tokens are wasteful and just terrible, at the input.

Maybe it makes more sense that all inputs to LLMs should only ever be images. Even if you happen to have pure text input, maybe you'd prefer to render it and then feed that in.

I dislike the tokenizer. Tokenizers are ugly, separate, not end-to-end stage. It "imports" all the ugliness of Unicode, byte encodings, it inherits a lot of historical baggage, security/jailbreak risk (e.g. continuation bytes). It makes two characters that look identical to the eye look as two completely different tokens internally in the network. A smiling emoji looks like a weird token, not an... actual smiling face, pixels and all, and all the transfer learning that brings along. The tokenizer must go.

https://github.com/deepseek-ai/DeepSeek-OCR

https://news.ycombinator.com/item?id=45652952

According to the DeepSeek paper, you can pull out 10 text tokens from a single image token with near-100% accuracy. In other words, a model’s internal representation of an image is ten times as efficient as its internal representation of text. Does this mean that models shouldn’t consume text at all? When I paste a few paragraphs into ChatGPT, would it be more efficient to convert that into an image of text before sending it to the model? Can we supply 10x or 20x more data to a model at inference time by supplying it as an image of text instead of text itself?

For all of the AI industry’s advancements, the major models like ChatGPT and Claude still struggle with PDFs, one of the oldest and ubiquitous file formats.

LLM’s suck at complex OCR, and probably will for a while. LLMs are excellent for many text-generation or summarization tasks, but they falter at the precise, detail-oriented job of OCR—especially when dealing with complicated layouts, unusual fonts, or tables. These models get lazy, often not following prompt instructions across hundreds of pages, failing to parse information, and “thinking” too much.

LLMs process images through high-dimensional embeddings, essentially creating abstract representations that prioritize semantic understanding over precise character recognition

Consider a simple table cell containing "1,234.56". The LLM might understand this represents a number in the thousands, but lose critical information about:

Exact decimal placement
Whether commas or periods are used as separators
Font characteristics indicating special meaning
Alignment within the cell (right-aligned for numbers, etc.)

https://news.ycombinator.com/item?id=42966958

Markdown extraction is just the first step. For documents to be effectively used in RAG pipelines, they must be split into smaller, semantically meaningful chunks.

Recent studies have shown that using large language models (LLMs) for this task can outperform other strategies in terms of retrieval accuracy. This intuitively makes sense - LLMs excel at understanding context and identifying natural boundaries in text, making them well-suited for generating semantically meaningful chunks.

The problem? Cost. Until now, LLM-based chunking has been prohibitively expensive. With Gemini Flash 2.0, however, the game changes again - it's pricing makes it feasible to use it to chunk documents at scale.

https://news.ycombinator.com/item?id=42952605

(disclaimer I am CEO of llamaindex, which includes LlamaParse)
Nice article! We're actively benchmarking Gemini 2.0 right now and if the results are as good as implied by this article, heck we'll adapt and improve upon it. Our goal (and in fact the reason our parser works so well) is to always use and stay on top of the latest SOTA models and tech :) - we blend LLM/VLM tech with best-in-class heuristic techniques.

Some quick notes: 1. I'm glad that LlamaParse is mentioned in the article, but it's not mentioned in the performance benchmarks. I'm pretty confident that our most accurate modes are at the top of the table benchmark - our stuff is pretty good.

2. There's a long tail of issues beyond just tables - this includes fonts, headers/footers, ability to recognize charts/images/form fields, and as other posters said, the ability to have fine-grained bounding boxes on the source elements. We've optimized our parser to tackle all of these modes, and we need proper benchmarks for that.

3. DIY'ing your own pipeline to run a VLM at scale to parse docs is surprisingly challenging. You need to orchestrate a robust system that can screenshot a bunch of pages at the right resolution (which can be quite slow), tune the prompts, and make sure you're obeying rate limits + can retry on failure.

Poppler is a PDF rendering library based on the xpdf-3.0 code base.

SVG to EMF Converter - CloudConvert is a free & fast online file conversion service.

https://graphicdesign.stackexchange.com/questions/60996/illustrator-emf-export-loses-precision

New, powerful features have recently been added to PDF::Builder and PDF::Table, enabling faster and easier high-level generation of PDF documents. The versions are respectively 3.025 and 1.005, and are available on CPAN.

As well as the ability to "pour" text into a document's defined page areas, and have it flow easily over pages, this new version also enables high level text formatting with markup languages, as well as much-enhanced font management. The markup supports Markdown (via Text::Markdown) and a large (and growing) subset of HTML/CSS, as well as simple paragraphs-only markup, and for the first time can be used to format cells in PDF::Table. This is far from the final version, as many improvements are in the pipeline for this functionality, and are expected to be released over the coming one to two years. These will include, among other things, proper word hyphenation and probably some form of paragraph shaping, such as Knuth-Plass. The full list (at this time) is at https://github.com/PhilterPaper/Perl-PDF-Builder/issues/195 .