Hello, Codeforces-style readers!

Anyone who has set a problem knows the statement is the easy part. After it come the validator, the checker, a main solution, some wrong solutions that must fail, tests that reach the limits, and finally uploading it all to Polygon by hand.

AutoSetter is our attempt to automate that. You give it a picture or PDF of a problem statement, and it gives back a complete Polygon package:

• problem.json: the statement read into a structured spec (title, story, input/output format, constraints, samples, limits)
• a statement, a C++17 main solution, plus greedy, brute-force and heavy solutions
• a testlib validator.cpp and checker.cpp
• tests generated with Z3 from a declarative spec, including max-size cases
• a validation report and a manifest.json saying whether the package is fit to release

Before generating anything, it checks a vector database of existing problems. If the problem already exists, you get the link and no package. Otherwise it can push the package to Polygon with your API key.

This page explains how the project works inside. It is not a product page.

One command runs every stage in order. Each stage writes its files to out/, so you can inspect any step.

Modules

When one model writes the validator, the solution and the checker from the same JSON, all three can share the same mistake and still pass each other. So the pipeline never trusts a file because another generated file agrees with it:

1. Official samples check the validator. The samples in the statement are known-good input, so a correct validator must accept all of them. If it rejects one, the validator (or the extracted constraints) is wrong.
2. The validator checks the generator. If the validator accepts the samples but rejects a generated test, the generator is the file at fault.
3. Wrong outputs probe the checker. Running the checker on the solution's own output only shows that x == x. So we also feed it outputs we know are wrong:

  ✅ checker rejected: an empty output file
  ✅ checker rejected: a correct answer with its last token missing
  ✅ checker rejected: a correct answer with an extra token appended
  ✅ checker rejected: a correct answer with every number increased by one

When a check fails, only the files at fault are regenerated, with the exact error as feedback, for up to three rounds.

Models write random generators that often break the constraints, for example k > n or a sum(n) over the limit. So instead the model writes a declarative spec of the input:

{ "vars": { "n": [1, 200000], "k": [1, "n"] },
  "arrays": { "a": { "len": "n", "range": [1, 1000000000] } },
  "file_constraints": ["sum(n) <= 200000"], ... }

• Z3 solves the integers. Each integer is a variable with its bounds and constraints, fixed one at a time inside the range Z3 says is still feasible.
• Builders fill the bulk. Arrays, strings, trees and graphs are built directly, so an n = 2·10⁵ test takes about a second.
• Each seed has a strategy: 1 = all minimum, 2 = all maximum, then random, small, near-max, log-scaled, with matching shapes (sorted arrays, path/star trees).
• The spec must parse the official samples. If it can't, it goes back to the model like a compile error.

vector_database/ collects Codeforces and LeetCode problems, normalizes and deduplicates them, embeds them with BAAI/bge-small-en-v1.5 (384 dimensions), and stores them in Qdrant with cosine distance.

At run time, the extracted problem.json is embedded the same way, and we take only its single nearest neighbour (k = 1). If the cosine similarity is above 0.80, we print the link and stop.

Results from a real run against our 12-problem test collection:

The last row is a known issue. This embedding model rates even unrelated problems around 0.75–0.80, while real copies score about 0.95, so a 0.80 threshold sits in the noise. It is configurable (--similarity-threshold), and around 0.90 looks safer.

The CLI asks for your Polygon API key and secret first and verifies them, so a long run never stops at the end to wait for input. Once the package passes validation, it creates the problem and uploads everything.

$ autosetter statement.png
Polygon API key (blank to skip upload): ••••••••
Polygon API secret:
Polygon credentials verified.
Generating JSON specification...
Searching vector database for an existing copy of this problem...
  0.8062  Given Length and Sum of Digits...
Generating statement... validator... generator... solution... checker...
  ✅ Validator accepts all 1 official sample(s)
  Test 1/10: ✅  ...  Test 10/10: ✅
✅ All 10 tests passed!
Ready for release — all 10 tests passed

Pushing package to Polygon...
Created Polygon problem 'two-sum' (ID 412345).
▶ Setting limits: 2000 ms, 256 MB...
▶ Uploading checker.cpp / validator.cpp / testlib.h / generator.cpp...
▶ Uploading solutions...  solution.cpp [MA] greedy [WA] brute [TL] heavy [TL]
▶ Uploading statement...
▶ Uploading tests...
▶ Committing changes...
▶ Requesting package build (with verification)...
Polygon problem: two-sum (ID 412345)

Transcript from a demo run with the models and Polygon replaced by fakes. The CLI, Qdrant search, compile and validation were real.

Exit codes

The project's main branch with its feature branches and pull requests. Each numbered dot is a milestone, listed below.

git log --graph (latest on main)

* fd25a5d polygon connected and vector db integrated readme updated
* dfc917f Add similar-problem search and fix vector database pipeline
*   14bee90 Merge pull request #5 from Jenmi07/main
|\
| * f0c5a5b image added
|/
* 7f3aaf8 Modified model name
* 005f53f Added test images
* 520784f Changed example images
* 2bcf73a Modified link to connect to server
*   1a24825 Merge pull request #4 from YajatGupta07/main
|\
| * bfd530d z3
|/
* 5321a8c coordinating commits
*   2a21004 Merge branch 'main' of github.com:devlup-labs/AutoSetter
|\
| * cda7ab3 pytest failure reset
| * c3a8b0a Rewrite generator.txt: full 15-phase algorithmic prompting
| * 4201eeb Add vector database pipeline by swasti_singhal
| *   c748008 Merge remote-tracking branch 'origin/main' into main
| |\
| * | 2a3cfb0 Z3 files update
* | | f5a3873 Changed ollama models and ollama host on config.py
| |/
|/|
* | 20245ee Merge branch 'main' of github.com:devlup-labs/AutoSetter
|\|
| * 231d43a colab script added
* | 23cafb2 modified prompts to algorithmic prompts
|/
* 8689148 sanitycheckdefined

Repository: devlup-labs/autosetter
Commit: fd25a5d
Files analyzed: 163
Estimated tokens: 149.5k   (vendored testlib.h and images excluded)

Directory structure:
└── devlup-labs-autosetter/
    ├── .github/workflows/tests.yml
    ├── autosetter/
    │   ├── include/testlib.h
    │   ├── prompts/
    │   │   ├── checker.txt
    │   │   ├── generator.txt
    │   │   ├── json_extraction.txt
    │   │   ├── solution.txt
    │   │   ├── solution_brute.txt
    │   │   ├── solution_greedy.txt
    │   │   ├── solution_heavy.txt
    │   │   ├── statement.txt
    │   │   ├── test_spec.txt
    │   │   └── validator.txt
    │   ├── testgen/
    │   │   ├── __init__.py
    │   │   ├── __main__.py
    │   │   ├── builders.py
    │   │   ├── engine.py
    │   │   ├── expr.py
    │   │   ├── layout.py
    │   │   └── spec.py
    │   ├── __init__.py
    │   ├── __main__.py
    │   ├── cli.py
    │   ├── config.py
    │   ├── extractor.py
    │   ├── generator.py
    │   ├── llm.py
    │   ├── packager.py
    │   ├── pipeline.py
    │   ├── polygon.py
    │   ├── prompts.py
    │   ├── remote.py
    │   ├── runner.py
    │   ├── sandbox.py
    │   ├── similarity.py
    │   └── vision.py
    ├── example/
    │   ├── 01_intake … 04_package/
    │   ├── README.md
    │   └── run_mock_pipeline.py
    ├── sandbox/
    │   ├── docker/Dockerfile
    │   ├── scripts/
    │   │   ├── build.sh
    │   │   ├── start.sh
    │   │   └── stop.sh
    │   └── server/
    │       ├── src/
    │       │   ├── config.js
    │       │   ├── executor.js
    │       │   ├── pool.js
    │       │   └── server.js
    │       └── package.json
    ├── tests/
    │   ├── conftest.py
    │   ├── fixtures.py
    │   ├── test_cli.py
    │   ├── test_extractor.py
    │   ├── test_generator.py
    │   ├── test_packager.py
    │   ├── test_pipeline.py
    │   ├── test_polygon.py
    │   ├── test_remote.py
    │   ├── test_sandbox.py
    │   ├── test_similarity.py
    │   ├── test_testgen.py
    │   └── test_vision.py
    ├── vector_database/
    │   ├── embeddings/
    │   │   ├── __init__.py
    │   │   └── embedder.py
    │   ├── processing/
    │   │   ├── __init__.py
    │   │   ├── deduplicate.py
    │   │   ├── normalize.py
    │   │   └── schema.py
    │   ├── qdrant/
    │   │   ├── __init__.py
    │   │   └── client.py
    │   ├── scripts/
    │   │   ├── collect.py
    │   │   ├── embed.py
    │   │   ├── normalize.py
    │   │   ├── run_pipeline.py
    │   │   ├── test_search.py
    │   │   └── upload_qdrant.py
    │   ├── sources/
    │   │   ├── __init__.py
    │   │   ├── base.py
    │   │   ├── codeforces.py
    │   │   └── leetcode.py
    │   ├── config.py
    │   ├── pipeline.py
    │   ├── README.md
    │   ├── requirements.txt
    │   └── search.py
    ├── app.py
    ├── ARCHITECTURE.md
    ├── LICENSE
    ├── pyproject.toml
    ├── pytest.ini
    ├── README.md
    └── requirements.txt

For the full text digest, paste the repo URL into gitingest.com, or run pip install gitingest && gitingest https://github.com/devlup-labs/AutoSetter.