How we rank tools

Each criterion is scored from 0 to 5, separately for every task a tool does. A tool's score for “Merge PDF” can differ from its score for “Compress PDF”.

That gives three rankings: the best tools for a task, a tool's overall score (the average of its scored tasks), and the best tools in a category.

A score is either tested (we ran it ourselves) or claimed (stated by the maker and not yet verified). Claimed values are shown but never counted in the overall score.

A task score is the weighted average of the tested criteria, using the percentages listed for each category above. If a criterion is not tested yet, the remaining weights are rescaled, so a missing test never counts as a zero. It appears only when at least 4 criteria have been tested. Below that, the page says “Not enough data”.

agathox makes some of the tools it lists, so we say exactly how they are tested.

Every test uses the same fixed input files in a real browser, and the files and the scripts are published in the repository (testing/), so anyone can rebuild them and get the same results.

Our own tools are tested automatically. Other makers' tools are never driven by a robot: their terms often forbid it, and bot protection makes the results unreliable. Instead a person uses the tool in a normal browser with the same files, and a script only watches: it counts the clicks, notes whether the page sent the file to a server, and opens the downloaded file to count its pages. Until a tool has been tested this way it shows “Not enough data”.

Automatic testing needs the owner's permission. The owner of any tool can allow us to test it automatically by publishing a small text file on their own site (/.well-known/agathox-test.txt containing the word allow). Only the owner can put a file there, so that is the permission we rely on. We check every day. When it is there, the tool is tested with the same files and scored; when it is not, we do not test it, and its page says so. If our tester cannot reach a clear result on a site, it scores nothing.

No ranking from one test. A list is put in score order only once at least 2 of its tools have been tested the same way. Until then the list is put in order by the signals rank below, and says so, so being the only tool tested never makes a tool look first.

PDF tools: how each score is set

• Correctness. 5 = the result opens and has exactly the expected pages (for Compress, also at least 20% smaller). 4 = Compress worked but saved under 20%. 3 = it opened but the tool kept the original unchanged. 2 = it opened with the wrong number of pages. 0 = no usable file.
• Privacy. 5 = we watched the page and no file data was sent anywhere. 3 = the file was uploaded, and the tool says it deletes it right after use. 1 = the file was uploaded and the tool says nothing about deleting it.
• Limits. Of our three test files (2.5, 12.6 and 26.4 MB): 4 = the 26.4 MB file worked. 3 = the 12.6 MB file worked. 2 = only the 2.5 MB file worked. 1 = none. We do not award 5, because we did not test anything larger.
• Ease. Counted actions from opening the page to downloading (choosing files counts as one, however it is done; cookie banners are not counted). 3 actions or fewer = 5, 4 = 4, 5 = 3, 6 = 2, more = 1. Needing an account caps it at 2.
• Free. 4 = the first use worked with no account, payment or prompt. We cannot see daily caps or hidden watermarks from one run, so we do not award 5. 2 = it asked for an account or payment, or the result carried a visible watermark, logo or extra page. 0 = it did not work.
• Speed. Not scored yet. Our runs are on a local copy, so a time would say nothing about what you would get over the internet.

The inputs: two small text PDFs (3 and 5 pages), a 2 file photo PDF built from high-quality JPEGs, and files of 12.6 and 26.4 MB. We use JPEGs because that is what photos in PDFs almost always are.

Many tools on agathox are only listed. We found them, visited their site to check that it answers, and kept the description the site gives about itself. We have not tested them or recorded facts about their limits, pricing or privacy.

Listed tools have no score, are never ranked above a tested tool, and are shown in a separate, alphabetical section marked “not yet tested”. A page made up only of listed tools is kept out of search engines. Makers can claim a listing, correct it and request a test.

The all-tools ranking mixes two scores: tested quality (70%) and real usage (30%).

Usage is monthly visitors and outbound clicks that agathox counts itself, shown on a log scale so one very large tool does not flatten the rest. Numbers a maker reports are shown but never counted, because they cannot be checked and clicks are easy to inflate.

A tool needs both a quality score and measured usage to get a combined score. Quality has the larger weight on purpose, so a popular but weaker tool cannot bury a better one.

Every tool with a website gets a second, separate number that a robot works out with no person involved. It says how reliable, secure, private and open the website is. It does not say how well the tool does its job; only a hands-on test can, and that is the “tested” score above. The two are never mixed together.

• Reliability30%

  Does the home page answer, and how fast does it finish loading? A site is only called down after it fails two checks in a row.

• Security25%

  Is it served over https, and how many of five protective headers does it set?

• Privacy25%

  How many third-party tracking services (analytics, ad pixels, session recorders) load on the page? Fewer is better.

• Transparency20%

  Does the site link a privacy policy and a pricing page?

Each part is scored from 1 to 5 by fixed rules and the evidence is printed next to it on the tool's page. The numbers are measured again every week.

A third, separate number comes from what people say about a tool in public discussions we are allowed to read. Today that is Hacker News, through its open search. We look only for stories that link to the tool's own website and whose title names the tool, so a news story that merely links a company's home page is not counted.

• Each commenter counts once. A comment counts only if it names the tool, and only the sentences around that mention are judged, because comments in a thread are often about something else.
• A simple word list (with “not good” counted as negative) sorts comments into positive or negative; neutral ones and questions are ignored.
• The score is the balance of positive and negative commenters, pulled toward 3 so a few comments cannot give an extreme result. It is shown only with at least 12 different commenters from at least 2 threads; otherwise the page says there is not enough discussion.
• Every score shows the threads and example quotes behind it with links, because this kind of reading can misjudge sarcasm. It is never mixed into the tested or measured scores.

We do not use X/Twitter, Trustpilot, G2 or Capterra, because their terms do not allow this. For tools with an open-source repository we also show GitHub stars, open issues and the last update, for information only.

Most lists have no hands-on test scores yet, so we still put them in order, using only information we can measure or quote without using the tool. We call this the signals rank. It is not a quality test and is never shown as one: it says how open, reliable and well documented a tool looks from the outside.

• Website health50%

  The measured score above (reliability, security, privacy, transparency), 1 to 5.

• Facts stated30%

  How many of the five facts (free plan, paid plan, signup, watermark, processing) the tool's own pages state clearly, from 1 to 5. Only for tools whose pages we have read.

• Community20%

  The community score above, 1 to 5, only when enough different people discussed the tool.

The score is the weighted average of the signals a tool has, averaged with a neutral 3.0 that counts as 30 points of weight. That means a tool measured on only one signal cannot outrank a tool with equally good results on all three, and no tool can reach 5.0 from signals alone. A tool with neither a health measurement nor any stated facts gets no signals score and is listed last. Each score says how many of the three signals it rests on.

The moment at least 2 tools in a list have hands-on test scores, the list switches to the tested order and the signals rank is no longer used there. Hands-on scores always come first.

To measure a site we request its home page once a week, after reading its robots.txt. We identify ourselves as agathox-bot. If you do not want to be measured, add these two lines to your robots.txt and we will stop: User-agent: agathox-bot and Disallow: /. Your tool is then shown without a measured score, and nothing else changes.

agathox makes some of the tools it lists. Those are labelled “Made by agathox” and scored by the same method as the rest.

Facts about other makers' tools (prices, limits, signup, where files go) come from their own official pages, with the date we read them and a link. Where a page does not say, we write “Not stated”. Names and trademarks belong to their owners, and agathox is not affiliated with them.

Paid placements, if we ever add them, will be clearly labelled and kept apart from the ranking. Money will not change a score or its position.

If a score about your tool is wrong, tell us here with the details and we will retest it.