While benchmarking /api/dataValueSets across versions I found a large
regression in the JSON export path between 2.42 and 2.43. CSV from the same runs
is barely affected, which is what isolates it. Posting here rather than Jira
because external accounts can’t create issues there.
Setup: Sierra Leone demo database, PostgreSQL 16, same host and container
limits for every run, curl to /dev/null, cold run discarded. Both versions
returned exactly 973,730 values for the same 24-month window, so this is
like-for-like.
| 2.42.5.2 | 2.43.1 | master (df80564) |
|
|---|---|---|---|
| JSON | 3.25 s | 23.79 s | 12.13 s |
| JSON, µs per value | 3.2 | 21.0 | 9.3 |
JSON + compression=gzip |
4.29 s | 51.17 s | 12.64 s |
| CSV | 3.77 s | 7.39 s | 7.14 s |
| CSV, µs per value | 3.7 | 4.3 | 3.8 |
Database time is 2.91 s and identical whichever format is requested
(pg_stat_statements, counters reset immediately before each request) — the
difference is entirely in serialisation. Worth noting that on 2.42 JSON was
faster than CSV despite emitting 2.7× more bytes; that ordering inverts in
2.43. It also reproduces on a structurally different dataset (Child Health,
1,169,186 values: 20.5 µs/value JSON against 3.8 for CSV).
Cause
2.43 replaced JsonDataValueSetWriter and friends with DataExportOutput on
top of org.hisp.dhis.jsontree.JsonBuilder. The 2.42 writer used Jackson with
bulk escaping and FLUSH_PASSED_TO_STREAM explicitly disabled.
2.43.1 pins json-tree 1.8.1 (dhis-2/pom.xml:99). In that version
JsonBuilder.streamObject wraps the output in a PrintStream, and escaping
runs one character at a time via str.chars().forEachOrdered(...). Every
character therefore goes through PrintStream.append(char), which is
synchronized and flushes both buffers on each call.
JFR (settings=profile, 2,141 execution samples on request threads) puts 60% of
samples in JsonAppender and 63% in stream/encoder machinery, against 8% in
JDBC and Hibernate. Top allocated class on request threads is
java.nio.HeapCharBuffer — one wrapper per character written.
The gzip number is the clearest signature: enabling compression sends 31× fewer
bytes over the wire and makes the request 27 s slower, because the
per-character flush drives Deflater.deflate once per character. Per MB of
input, compressing the JSON costs 0.094–0.108 s against 0.004 s for CSV — a 25×
difference for the same algorithm at the same level.
Status on master
master pins json-tree 1.9.4 (dhis-2/pom.xml:104). 1.9.0 replaced the
PrintStream in streamObject with an OutputStreamWriter, removing the
per-character flush. That recovers serialisation from 21.0 to 9.3 µs/value and
collapses the gzip penalty from +27.4 s to +0.5 s — strong confirmation that the
flush was the mechanism.
It does not get back to 2.42’s 3.2 µs/value, because escaping is still
per-character in 1.9.4. I’ve filed that separately against the library:
As of 2026-08-18 the 2.43 branch still pins 1.8.1, so released instances
have none of this.
The question
Would you consider backporting the json-tree bump to the 2.43 branch? The
improvement is measured rather than assumed, and 2.43 is what implementations
are running now.
Workaround for anyone on 2.43 today
dataValueSets.csv with ?compression=gzip — 7.94 s and 6.1 MiB against JSON’s
23.79 s and 246.4 MiB. Note that compression is a query parameter; an
Accept-Encoding header has no effect. This is a workaround for a 2.43 defect,
not general advice — on 2.42 JSON was the faster format.
Caveats
master differs from 2.43 by months of work, so the improvement can’t be
attributed to the dependency bump alone — the direction and size are measured,
the cause is inferred from source plus the gzip signature. master wasn’t
profiled. Run-to-run spread on the 24-month JSON figure is about 1.4 s.
Happy to share the full measurement set, JFR recording or the exact request
URLs if that’s useful.