ISO/IEC 15938-18:2023

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 15938-18
ISO/IEC JTC 1/SC 29 Secretariat: JISC
Voting begins on: Voting terminates on:
2022-07-25 2022-10-17
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
ICS: 35.040.40
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 15938-18:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022

---------------------- Page: 1 ----------------------
ISO/IEC DIS 15938-18:2022(E)
DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 15938-18
ISO/IEC JTC 1/SC 29 Secretariat: JISC
Voting begins on: Voting terminates on:

Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
ICS: 35.040.40
COPYRIGHT PROTECTED DOCUMENT
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
© ISO/IEC 2022
THEREFORE SUBJECT TO CHANGE AND MAY
This document is circulated as received from the committee secretariat.
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
NOT BE REFERRED TO AS AN INTERNATIONAL
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on STANDARD UNTIL PUBLISHED AS SUCH.
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
IN ADDITION TO THEIR EVALUATION AS
or ISO’s member body in the country of the requester. BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
ISO copyright office
USER PURPOSES, DRAFT INTERNATIONAL
CP 401 • Ch. de Blandonnet 8
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
CH-1214 Vernier, Geneva
POTENTIAL TO BECOME STANDARDS TO
Phone: +41 22 749 01 11
WHICH REFERENCE MAY BE MADE IN
Reference number
Email: copyright@iso.org
NATIONAL REGULATIONS.
Website: www.iso.org ISO/IEC DIS 15938-18:2022(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
Published in Switzerland
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
ii
  © ISO/IEC 2022 – All rights reserved
PROVIDE SUPPORTING DOCUMENTATION. © ISO/IEC 2022

---------------------- Page: 2 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance testing . .2
4.1 General . 2
4.2 Conformance testing for decoder . 2
4.3 Conformance testing for bitstreams . 2
4.4 Models and reference bitstreams . 2
4.5 Procedure to test decoders . 7
4.5.1 Decoding self-contained NNR bitstreams . 7
4.5.2 Decoding NNR bitstreams using out-of-band parameters . 7
4.6 Procedure to test bitstreams . 7
5 Reference software . 7
5.1 Software location and license . 7
5.2 Software installation . 8
5.3 Software architecture . 8
5.3.1 General . 8
5.3.2 Parameter reduction methods . 9
5.3.3 Parameter approximation . 9
5.3.4 Reconstruction . . . 10
5.3.5 Encode. 10
5.3.6 Decode . 10
5.4 Data structures and interfaces . 10
5.4.1 model_info – Shared model information. 10
5.4.2 approx_data – Data structure for interface #4 .12
5.4.3 nctm – Main module .13
5.4.4 nctm.nnr_model – Module for handling model related functionalities . 16
Annex A (informative) Implementation in Python .20
Bibliography .21
iii
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 3 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO website and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
iv
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Introduction
This document describes conformance testing and the reference software for ISO/IEC 15938-17
compression of neural networks for multimedia description and analysis. The reference software
includes both encoder and decoder functionality.
The reference software is useful in aiding users of a standard for coding neural networks to establish
and test conformance and interoperability, and to educate users and demonstrate the capabilities of
the standard. For these purposes, the accompanying software is provided as an aid for the study and
implementation of 15938-17 compression of neural networks for multimedia description and analysis.
The purpose of International Standard is to provide the following:
— A set of reference bitstreams conforming to ISO/IEC 15938-17.
— Description of procedures to test conformance of bitstreams and decoders to ISO/IEC 15938-17.
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
v
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 5 ----------------------
DRAFT INTERNATIONAL STANDARD ISO/IEC DIS 15938-18:2022(E)
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
1 Scope
This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and
provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 as an
electronic attachment. The software is an integral part of this International Standard.
Requirements established in ISO/IEC 15938-17 take precedence over the behaviour of the reference
software. The use of this reference software is not required for making an implementation of an
encoder or decoder in conformance to ISO/IEC 15938-17. The reference software is however required
for the conformance testing procedures of decoders and bitstreams.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-17:2022, Information technology — Multimedia content description interface — Part 17:
Compression of neural networks for multimedia content description and analysis
ISO/IEC 21778, Information technology — The JSON data interchange syntax
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-17 and the
following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
array
ordered list of elements where all elements are of the same type
3.2
dictionary
ordered list of key/value pairs where each key/value pair is an list of two elements with the first element
being denoted ‘key’ and with the second element being denoted ‘value’
1
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 6 ----------------------
ISO/IEC DIS 15938-18:2022(E)
4 Conformance testing
4.1 General
The following subclauses specify normative tests for verifying the conformance of bitstreams as well as
decoders. Those normative tests make use of test data (bitstreams and related models) provided as an
electronic attachment to this document (described in detail in subclause 4.4) and follow the procedure
described in subclause 4.5.
4.2 Conformance testing for decoder
The decoder conformance of ISO/IEC 15938-17 is specified in clause 7 of that document.
4.3 Conformance testing for bitstreams
The bitstream conformance of ISO/IEC 15938-17 is specified in clause 7 of that document.
4.4 Models and reference bitstreams
A set of bitstreams and related neural network models is provided for conformance testing. When neural
network models are provided, they are the source models used to generate one or more compressed
bitstreams. The code defining these models is provided as part of the reference software distribution. A
bitstream may be generated
— from a commonly used neural network model, e.g., MobileNet
— from synthetic data by encoding it with the reference encoder
— by creating a conformant synthetic bitstream directly, without using an encoder
Where applicable, the dataset used to train the model is provided for information only. However, the
dataset is not needed for conformance testing and thus not provided.
Table 1 summarizes the provided bitstreams. It also lists the features tested with the bitstreams, and
the reference encoder configuration used to generate the bitstream (where applicable). If decoding the
bitstream requires out-of-band parameters (e.g., information that is derived from the network topology
description), those parameters are also provided.
Table 1 — Bitstreams and related models for conformance testing.
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance map scan_order = 1
MobileNetV2. approx_method = "codebook"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
2
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase.nctm mance map scan_order = 1
approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B.nctm and Pruning Per- scan_order = 1
formance map approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16. Net Performance Map scan_order = 1
nctm (pruned model) approx_method = "codebook"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topology -
encode_tpl_only = True
sparse_bm_ sparse bitmask
partial_data_counter = 0
DCase.nctm
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topology -
encode_tpl_only = True
prune_bm_ Net prune bitmask
partial_data_counter = 0
VGG16.nctm
prune_tpl_cont_ VGG16 Image 6.3.4.5 Prune Topolo-
encode_tpl_only = True
comb_bm_ Net gy - combined
partial_data_counter = 0
VGG16.nctm bitmask
prune_tpl_cont_ DCase DCase 6.3.4.5 Prune Topology - encode_tpl_only = True
prune_dictionary_ prune dictionary topology_indexed_reference_
DCase.nctm flag = False
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
3
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
prune_tpl_cont_ ResNet50 Image 6.3.4.5 Prune Topology - encode_tpl_only = True
prune_ Net prune dictionary topology_indexed_reference_
dictionary_idx_ (indexed elem id) flag = True
ResNet50.nctm partial_data_counter = 0
tpl_reflist_DCase. DCase DCase 6.3.4.5, Topology Reflist encode_tpl_only = True
nctm 6.3.3.7 partial_data_counter = 0
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_65536. 65536
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_32768. 32768
nctm
partial_data_ VGG16 Image 6.3.3.1 Partial data
counter_VGG16_ Net counter max_ndu_nnr_unit_size =
ndu_size_16384. 16384
nctm
partial_data_ DCase DCase 6.3.3.1 Partial data
counter_DCase_ counter max_ndu_nnr_unit_size =
ndu_size_2048. 2048
nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
ter_DCase_ndu_ counter
1024
size_1024.nctm
deepCABAC_ ResNet50 Image 10 DeepCABAC en- see verify_all.sh
ResNet50_1_qp-38_ Net 9.1.1 / 9.2.1 tropy coding,
qp_density2.nctm uniform quanti-
zation
dependent_ ResNet50 Image 9.1.3 / 9.2.3 Dependent scalar see verify_all.sh
quantization_ Net 6.3.3.7 quantization
ResNet50_2_qp-38_
qp_density2.nctm
deepCABAC_qp_ MobileNetV2 Image 9.2 QpDensity see verify_all.sh
density_Mobile- Net
NetV2_3_qp-38_qp_
density2.nctm
deepCABAC_ MobileNetV2 Image 9.2 QpDensity see verify_all.sh
qp_density_ Net
MobileNetV2_4_
qp-76_qp_density3.
nctm
block_scan_ ResNet50 Image 4.12 / 6.4.3.7 Block scan order see verify_all.sh
order_8x8_cabac_ Net / 6.4.3.8 / / cabac entry
entry_points_ 7.3.6 points
ResNet50_5_qp-38_
qp_density2.nctm
block_scan_or- ResNet50 Image 4.12 / 6.4.3.7 Block scan order see verify_all.sh
der_16x16_cabac_ Net / 6.4.3.8 / / cabac entry
entry_points_ 7.3.6 points
ResNet50_6_qp-38_
qp_density2.nctm
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
4
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
codebook_signal- MobileNetV2 Image 9.1.3 / 9.2.2 Codebook-based see verify_all.sh
ing_MobileNetV2_7_ Net quantization
qMse0.00001.nctm
local_scaling_ DCase DCase 8.2.7 / 8.3.7 Local scaling see verify_all.sh
DCase_8_qp-38_qp_
density2.nctm
batchnorm_ MobileNetV2 Image 8.2.6 / 8.3.6 BatchNorm see verify_all.sh
folding_Mobile Net Folding
NetV2_9_qp-38_qp_
density2.nctm
out_of_band_ ResNet50 Image 6.3.3.7 / Out-of-band sign- see verify_all.sh
2
signaling_Res Net 6.4.3.7 aling
Net50_10_qp-38_
qp_density2.nctm
deepCABAC_8bit_ resnet50 ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
ResNet50_PYT- (zoo) tization with
zoo_11_qp0_qp_ limited precision
density4.pt.nctm (8bit)
deepCABAC_8bit_ mobilenet_v2 ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
MobileNetV2_PY- (zoo) tization with
Tzoo_12_qp0_qp_ limited precision
density4.pt.nctm (8bit)
deepCABAC_4bit_ vgg16 (zoo) ImageNet 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
VGG16_PYTzoo_13_ tization with
qp0_qp_density4. limited precision
pt.nctm (4bit)
deepCABAC_8bit_ UC12B CIFAR- 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
UC12B_14_qp0_qp_ 100 tization with
density4.nctm limited precision
(8bit)
deepCABAC_4bit_ DCase DCase 9.1.1 / 9.2.1 Uniform quan- see verify_all.sh
DCase_15_qp0_qp_ tization with
density4.nctm limited precision
(4bit)
perf_map_ MobileNetV2 Image 6.3.4.3 sparsification qp_density = 2
sparse_ Net performance map scan_order = 1
MobileNetV2_bw8. (8bit) approx_method = "uniform"
nctm qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
5
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC DIS 15938-18:2022(E)
Table 1 (continued)
Bitstream id Source Data-set Relevant Features tested Reference encoder config-
1
model technology uration
perf_map_prune_ DCase DCase 6.3.4.3 pruning perfor- qp_density = 2
DCase_bw8.nctm mance map (8bit) scan_order = 1
approx_method = "uniform"
qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_sparse_ UC12B 6.3.4.3 Sparsification qp_density = 2
prune_UC12B_bw8. and Pruning scan_order = 1
nctm Performance map approx_method = "uniform"
(8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
perf_map_ VGG16 Image 6.3.4.3 Sparsification qp_density = 2
sparse_VGG16_bw8. Net Performance Map scan_order = 1
nctm (pruned model) approx_method = "uniform"
(8bit) qp = 35
qp_density = 2
opt_qp = False
disable_dq = True
lambda_scale = 0.0
cb_size_ratio = 5000
q_mse = 0.00001
param_opt_flag = False
cabac_unary_length = 10
partial_data_counter = 0
partial_data_ DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
counter_DCase_ counter (8bit)
2048
ndu_size_2048_
approx_method = "uniform"
bw8.nctm
partial_data_coun- DCase DCase 6.3.3.1 Partial data
max_ndu_nnr_unit_size =
ter_DCase_ndu_ counter (8bit)
1024
size_1024_bw8.
approx_method = "uniform"
nctm
1
This columns specifies the applicable clause in ISO/IEC 15938-17.
2
As specified in out_of_band_signaling_ResNet50_10_qp-38_qp_density2_oob.json
The bitstreams and models are provided at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The parameters of the bitstreams, for which no detailed parameters are listed, can be obtained from
the scripts given in column “Reference encoder configuration”, e.g., verify_all.sh in “run/conformance_
bitstreams” of the reference software. Furthermore, it also provides detailed reference software
configurations for each test. If out-of-band parameters are required, they are provided in a file
conforming to JSON (ISO/IEC 21778), specified in a footnote.
6
  © ISO/IEC 2022 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC DIS 15938-18:2022(E)
4.5 Procedure to test decoders
For testing a decoder, all bitstreams that cover mandatory features as specified in ISO/IEC 15938-17
shall be tested. If support for optional features is claimed, also the bitstreams covering these features
shall be tested.
On platforms without floating point support, or limited integer precision support, only those bitstreams
indicated as "(8bit)" or "(4bit)" shall be tested. On platforms with floating point support, testing these
limited precision bitstreams is optional.
For each bitstream, the procedure includes the following steps:
1. Decode bitstream B with the reference decoder to obtain the decoded model R according to
r
subclause 4.5.1 if no out-of-band parameters are specified and according to subclause 4.5.2,
otherwise,
2. Decode bitstream B with the decoder under test to obtain the decoded model R according to
t
subclause 4.5.1 if no out-of-band parameters are specified and according to subclause 4.5.2,
otherwise.
3. Compare every tensor in R against the corresponding tensor in R . The test shall be considered as
r t
passed if the following conditions hold for every pair of tensors:
— If an integer tensor has been obtained as a result (decoding method specified in clause 7.3.2 of
ISO/IEC 15938-17), the tensors shall be identical.
— If a tensor has been obtained as the result of another decoding method, no corresponding pair
of floating point values shall have an absolute difference larger than 0,000001.
4.5.1 Decoding self-contained NNR bitstreams
For self-contained bitstreams, the decoder may not use any other inputs than the bitstream to be
decoded.
4.5.2 Decoding NNR bitstreams using out-of-band parameters
The testing of decoding bitstreams using out-of-band parameters is agnostic to whether those are
application provided or decoded from a particular third party topology format. The decoder shall use
only the input bitstream and the parameters provided for this bitstream as specified in Table 1 as
inputs.
4.6 Procedure to test bitstreams
In order to test bitstreams created with an encoder other than the reference encoder, the bitstream
shall be decoded using the reference decoder. The bitstream is considered conformant, if no errors
occur during the decoding process.
All bitstreams that have been created with the script “verify_all.sh” specified in Table 1 can be decoded
with the script “decode_all.sh” in “run/conformance_bitstreams” of the reference software. The
bitstreams are considered conformant, if no errors occur during the decoding process.
5 Reference software
5.1 Software location and license
The reference software is available at https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en/ .
The software license is provided in the file LICENSE included with the software.
7
© ISO/IEC 2022 – All rights reserved

---------------------- Page: 12 ----------------------
ISO/IEC DIS 15938-18:2022(E)
The purpose of this clause is to provide the following:
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
Some examples of uses that may be appropriate for the reference decoder software are as follows:
— As an illustration of how to perform the decoding process specified in ISO/IEC 15938-17.
— As the starting basis for the implementation of a decoder that conforms to ISO/IEC 15938-17.
— For testing the conformance of a decoder implementation with the decoding process specified in
ISO/IEC 15938-17.
— For testing the conformance of a bitstream to the constraints specified for bitstream conformance
in ISO/IEC 15938-
...