# dataValueSets JSON export is ~7× slower in 2.43 than 2.42 — cause identified, fix exists upstream but is not backported **URL:** \ **Category:** Development - Développement\ **Created:** [18 August 2026 19:48 UTC](https://community.dhis2.org/t/datavaluesets-json-export-is-7x-slower-in-2-43-than-2-42-cause-identified-fix-exists-upstream-but-is-not-backported/76144 "2026-08-18T19:48:57Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![minogin](https://dhis2.b-cdn.net/user_avatar/community.dhis2.org/minogin/32/50280_2.png) [@minogin](https://community.dhis2.org/u/minogin)\ **Post date:** [18 August 2026 19:48 UTC](https://community.dhis2.org/t/datavaluesets-json-export-is-7x-slower-in-2-43-than-2-42-cause-identified-fix-exists-upstream-but-is-not-backported/76144/1 "2026-08-18T19:48:58Z")
While benchmarking `/api/dataValueSets` across versions I found a large regression in the JSON export path between 2.42 and 2.43. CSV from the same runs is barely affected, which is what isolates it. Posting here rather than Jira because external accounts can’t create issues there. **Setup:** Sierra Leone demo database, PostgreSQL 16, same host and container limits for every run, `curl` to `/dev/null`, cold run discarded. Both versions returned exactly **973,730 values** for the same 24-month window, so this is like-for-like. | | 2.42.5.2 | 2.43.1 | master (`df80564`) | | --- | --- | --- | --- | | JSON | **3.25 s** | 23.79 s | 12.13 s | | JSON, µs per value | 3.2 | 21.0 | 9.3 | | JSON + `compression=gzip` | 4.29 s | **51.17 s** | 12.64 s | | CSV | 3.77 s | 7.39 s | 7.14 s | | CSV, µs per value | 3.7 | 4.3 | 3.8 | Database time is 2.91 s and identical whichever format is requested (`pg_stat_statements`, counters reset immediately before each request) — the difference is entirely in serialisation. Worth noting that on 2.42 JSON was _faster_ than CSV despite emitting 2.7× more bytes; that ordering inverts in 2.43. It also reproduces on a structurally different dataset (Child Health, 1,169,186 values: 20.5 µs/value JSON against 3.8 for CSV). ### Cause 2.43 replaced `JsonDataValueSetWriter` and friends with `DataExportOutput` on top of `org.hisp.dhis.jsontree.JsonBuilder`. The 2.42 writer used Jackson with bulk escaping and `FLUSH_PASSED_TO_STREAM` explicitly disabled. 2.43.1 pins **json-tree 1.8.1** (`dhis-2/pom.xml:99`). In that version `JsonBuilder.streamObject` wraps the output in a `PrintStream`, and escaping runs one character at a time via `str.chars().forEachOrdered(...)`. Every character therefore goes through `PrintStream.append(char)`, which is `synchronized` and flushes both buffers on each call. JFR (`settings=profile`, 2,141 execution samples on request threads) puts 60% of samples in `JsonAppender` and 63% in stream/encoder machinery, against 8% in JDBC and Hibernate. Top allocated class on request threads is `java.nio.HeapCharBuffer` — one wrapper per character written. The gzip number is the clearest signature: enabling compression sends 31× fewer bytes over the wire and makes the request **27 s slower** , because the per-character flush drives `Deflater.deflate` once per character. Per MB of input, compressing the JSON costs 0.094–0.108 s against 0.004 s for CSV — a 25× difference for the same algorithm at the same level. ### Status on master master pins **json-tree 1.9.4** (`dhis-2/pom.xml:104`). 1.9.0 replaced the `PrintStream` in `streamObject` with an `OutputStreamWriter`, removing the per-character flush. That recovers serialisation from 21.0 to 9.3 µs/value and collapses the gzip penalty from +27.4 s to +0.5 s — strong confirmation that the flush was the mechanism. It does not get back to 2.42’s 3.2 µs/value, because escaping is still per-character in 1.9.4. I’ve filed that separately against the library: > > > \`JsonAppender.appendEscaped(CharSequence)\` dispatches once per character: > > \`\`\`ja…va > str.chars().forEachOrdered(this::appendEscaped); > \`\`\` > > (v1.9.4, \`JsonAppender.java:87\` — unchanged since at least 1.8.1, where it is line 107.) > > Each character goes through an \`IntStream\` element dispatch, a lambda call, a > \`switch\`, and a single-character write to the sink. For payloads that are > overwhelmingly plain ASCII this is all avoidable work: the strings in question > are 11-character alphanumeric UIDs, \`yyyyMM\` periods, ISO dates and numeric > values, none of which need escaping at all. > > \*\*Measured impact (via DHIS2 2.43/master, \`GET /api/dataValueSets.json\`, > 973,730 values, PostgreSQL 16, same host and configuration throughout):\*\* > > | | serialisation cost | > |---|---| > | DHIS2 2.42 (Jackson streaming writer, bulk escaping) | 3.2 µs/value | > | DHIS2 2.43.1 (json-tree 1.8.1) | 21.0 µs/value | > | DHIS2 master (json-tree 1.9.4) | 9.3 µs/value | > > 1.9.0 removed the \`PrintStream\` from \`JsonBuilder.streamObject\` in favour of > \`OutputStreamWriter\`, which removed a per-character flush and accounts for most > of the 21.0 → 9.3 improvement. The remaining gap to the Jackson-based writer is > consistent with the per-character dispatch above still being present. > > \*\*Suggested changes, roughly in order of value per unit of risk:\*\* > > 1. \*\*Bulk fast path.\*\* Scan the \`CharSequence\` for characters that need > escaping; when there are none — the common case — append the whole sequence > in a single call. \`Appender.of(Appendable)\` already overrides > \`append(CharSequence, int, int)\` and delegates whole ranges, so the sink can > take it. When escaping is needed, append the unescaped runs between escapes > rather than character by character. > 2. \*\*Indexed loop instead of \`chars().forEachOrdered(...)\`\*\*, removing the > \`IntStream\` allocation and per-element lambda dispatch. > 3. \*\*128-entry lookup table\*\* for "does this character need escaping", so the > common path is one array read rather than a \`switch\`. > 4. Optionally, an ASCII byte fast path that writes bytes directly rather than > chars through a charset encoder. > > \*\*Context for prioritisation:\*\* this path is used by DHIS2 for bulk data > export, where a single response can be hundreds of MB. Happy to supply the full > measurement set if useful. As of 2026-08-18 the **`2.43` branch still pins 1.8.1** , so released instances have none of this. ### The question Would you consider backporting the json-tree bump to the 2.43 branch? The improvement is measured rather than assumed, and 2.43 is what implementations are running now. ### Workaround for anyone on 2.43 today `dataValueSets.csv` with `?compression=gzip` — 7.94 s and 6.1 MiB against JSON’s 23.79 s and 246.4 MiB. Note that compression is a query parameter; an `Accept-Encoding` header has no effect. This is a workaround for a 2.43 defect, not general advice — on 2.42 JSON was the faster format. ### Caveats master differs from 2.43 by months of work, so the improvement can’t be attributed to the dependency bump alone — the direction and size are measured, the cause is inferred from source plus the gzip signature. master wasn’t profiled. Run-to-run spread on the 24-month JSON figure is about 1.4 s. Happy to share the full measurement set, JFR recording or the exact request URLs if that’s useful. ---
**Author:** ![jason](https://dhis2.b-cdn.net/user_avatar/community.dhis2.org/jason/32/32739_2.png) [@jason](https://community.dhis2.org/u/jason)\ **Post date:** [14 September 2026 10:59 UTC](https://community.dhis2.org/t/datavaluesets-json-export-is-7x-slower-in-2-43-than-2-42-cause-identified-fix-exists-upstream-but-is-not-backported/76144/2 "2026-09-14T10:59:26Z")
Thanks for the report @minogin . Could you provide a bit more detail as to exact request which you made? This will help us to reproduce your numbers. ---
**Author:** ![JanBernitt](https://dhis2.b-cdn.net/user_avatar/community.dhis2.org/janbernitt/32/30594_2.png) [@JanBernitt](https://community.dhis2.org/u/JanBernitt)\ **Post date:** [17 September 2026 09:58 UTC](https://community.dhis2.org/t/datavaluesets-json-export-is-7x-slower-in-2-43-than-2-42-cause-identified-fix-exists-upstream-but-is-not-backported/76144/3 "2026-09-17T09:58:56Z")
Thanks Andrey. I just got around to reading your report and started looking into it. The issues wasn’t on my radar as quite so dramatic but your report sounds plausible to me. Thanks for taking the time to run the tests and file a comprehensive issue. Will see what I can do about it. ---
**Author:** ![jason](https://dhis2.b-cdn.net/user_avatar/community.dhis2.org/jason/32/32739_2.png) [@jason](https://community.dhis2.org/u/jason)\ **Post date:** [18 September 2026 06:53 UTC](https://community.dhis2.org/t/datavaluesets-json-export-is-7x-slower-in-2-43-than-2-42-cause-identified-fix-exists-upstream-but-is-not-backported/76144/4 "2026-09-18T06:53:10Z")
Thanks for the report @minogin . I have been able to reproduce and trace exactly the problem you describe on 2.43. @JanBernitt is working on the JSONtree part, and we have also uncovered several other problems which have fixes in the pipeline. We will report back once things get fixed.