<!-- vim: set tw=78 sts=2 ai expandtab: -->
# debian-reference (Version 2 series)
2023-12-18T05:44:43+00:00
Osamu Aoki, <osamu at debian dot org>
This Debian Reference (v2) is intended to provide a broad overview of the
Debian system as a post-installation user's guide. It covers many aspects of
system administration through shell-command examples for non-developers.
## Package status
The package status of `debian-reference` can be checked at the following URLs:
* https://bugs.debian.org/src:debian-reference (BTS)
* https://tracker.debian.org/pkg/debian-reference (Package)
* https://www.debian.org/doc/manuals/debian-reference/ (Latest HTML page)
* https://www.debian.org/doc/user-manuals#quick-reference (Links on debian.org)
## Bug report
For bug report, please use the `reportbug` command against the
`debian-reference` source package.
```
$ reportbug --src debian-reference
```
For bug reports, it is strongly recommended to write what you want explicitly
as an alternative. "It does not work...", "I can do this...", etc. aren't
very useful. Bug report such as "The section 9.3.3 should be rewritten as
.... because .... " is desirable style. (I do not wish to spend time to
second-guess or search where you see the issues.)
For simple grammar errors and spell errors, diff -u against the plain text
file or the source file are most appreciated. (Of course, real patch files or
merge requests are even better.)
See also, http://bugs.debian.org for how Debian BTS works.
## Contributions
If you have a concrete update proposal, patch files posted to the Debian BTS
or merge requests for your proposal are highly appreciated.
You are welcomed to join the maintenance team and update the source tree.
For the English portion and the build scripts, please contact us with your
intended action at the Debian documentation project
<debian-doc@lists.debian.org> or file bug fix patches to the Debian BTS before
committing the changes to minimize major breakage.
For translation via PO files, it is nice if you check their correctness before
committing the changes. In practice, I have been quite permissive for the
translation updates and have fixed them manually if I find problems later.
All Debian Developers (DD) have the write access with the "Master" status for
this [debian-reference salsa repository](https://salsa.debian.org/debian/debian-reference).
If you are a non-Debian Developer
with `*-guest` account on salsa who needs write access to this
debian-reference salsa repository, you can inform your `*-guest` account to me
(or any DD) with clear intent and reasonable rationale and let me (or any DD)
invite you to the write access with the "Developer" status from
[the main debian-reference project page](https://salsa.debian.org/debian/debian-reference)
via Setting-Member menu.
See https://wiki.debian.org/Salsa/Doc for how to use this site.
## Source tree (main `salsa` repo)
Please check following web page and download the source tree using the `git`
command from salsa.debian.org.
* https://salsa.debian.org/debian/debian-reference
For read-only access:
```sh
$ git clone -o salsa https://salsa.debian.org/debian/debian-reference.git
$ git config merge.merge-po-files.driver "./bin/merge-po-files %A %O %B %P"
```
For read-write access with SSH:
```sh
$ git clone -o salsa git@salsa.debian.org:debian/debian-reference.git
$ git config merge.merge-po-files.driver "./bin/merge-po-files %A %O %B %P"
```
## Source tree (Debian package)
The latest released source package can be downloaded by:
```sh
$ apt-get source debian-reference
```
Here, `deb-src` line in `/etc/apt/sources.list` file should be pointing to
unstable.
## Source tree (alternative `weblate` repo)
Recently, people also seems to be contributing translation via the weblate
service.
* https://hosted.weblate.org/projects/debian-reference/
The git repo of debian-reference in weblate is:
* https://hosted.weblate.org/git/debian-reference/translations/
This service is good for translation only contributors and their translation can be
performed only via the web page GUI interaction without using `git` etc.
The translation result stays on `weblate` service and someone needs to merge
them to `salsa`.
Since there is no sanity check of entity etc. on this contribution, the person
merging from `weblate` to `salsa` should take care sanity check of the content.
The weblate service of debian-reference is currently mainly maintained by
xiao sheng wen <atzlinux@sina.com>.
## Updating `salsa` git repository
### Note on updating English contents and build setups
For English, please make sure to update and commit xml files in `rawxml/`.
If you updated other files, please make sure to update and commit them.
You must make a [test build of this source](#test-build-tips) before pushing it
to `salsa`.
If strings in po files needs to be updated together with ones in xml files in `rawxml/`, please
consider to edit and run `bin/sedpoxml`.
### Note on updating translation contents
If you are updating the translation only, please only commit the updated
`po/*.po` to the `latest` branch after [testing its
sanity](#sanity-check-of-a-po-file). I mean no commit of `po/debian-reference.pot`,
`*.xml`, and `*.ent` files by the translation focused users.
In other word, please **don't** use "`git commit -a`".
It is highly desirable to make a [sanity check of translation PO
files](#sanity-check-of-a-po-file) before committing the translation.
Of course, the best practice is to make a [test build of this
source](#test-build-tips) before committing the translation.
### Git operations for translation updates
Please note that the merging of PO files are non-trivial task which sometimes
doesn't work easily with the `git` merging infrastructure.
If you want to work on your current local branch `latest` without pushing to the remote
server, please fetch the remote branch `origin/latest` and rebase your
local branch `latest` to `origin/latest` to maintain the clean commit history.
Let's assume you are updating "fr":
```sh
$ git checkout latest
$ make wrap LANGPO=fr; make po LANGPO=fr # check fuzzy.log
... hack-hack-hack
$ make wrap LANGPO=fr; make po LANGPO=fr # check fuzzy.log
$ make check LANGPO=fr
... check sanity of PO file (optional)
$ make test LANGPO=fr
... check build/ for good HTML build
$ git add po/fr.po
$ git commit po/fr.po -m "PO: <fr updates"
$ git remote update origin # get origin/latest
$ gitk --all # check no remote updates
$ git push origin latest
```
Here, `make check` is an optional step. This can help to identify the error
location in the PO file easier than tracing errors in the test HTML build. If
you intentionally add supplemental contents with extra XML tags, you can add a
translator comment "`skip-tag-match`" for each such PO entry.
If merging is more complicated, you can create a local topic branch and work.
You can use such a topic branch as a remote backup, too.
Cherry-picking via `gitk --all` GUI is quite useful to resolve conflicts.
`git ime` (after installing `imediff` package may be useful splitting
complicated commits.
Any missing PO strings of `zh-tw`/`zh-cn` are temporarily auto-converted from
the existing `zh-cn`/`zh-tw` PO strings during the build time using opencc.
So you may find non-optimal translation strings not-found in the source tree
to show up in the generated document. I chose this build setting since
non-optimal strings are better than the original English ones for most Chinese
readers. You can have the correct translation strings by adding proper PO
strings.
Due to bugs in the TeX tool chain for the PDF generation, generation of the
proper PDF files may be disabled and only the placeholder PDF files may be
generated for some locales.
### Editor operations for translation updates
Translator are encouraged to use the dedicated PO file editors such as
`poedit`, `gtranslator`, `gettext.el` (with `emacs`), and `lokalize` on the PO
files `po/*.po` after obtaining the source tree with the `git` command.
The use of the dedicated PO file editor helps translator to find missing and
fuzzy translation data.
## Work with 2 remote services `salsa` and `weblate`
This is a bit complicated since data on 2 remote services `salsa` and `weblate`
may be altered simultaneously.
~~You need to merge data on 2 remote services and your local changes and update
them with merged results.~~
~~For this, `bin/merge-po-files` is used to give best merge result now.~~
~~Please make sure to update your local repository by running the following:~~
<!--
```sh
$ git config merge.merge-po-files.driver "./bin/merge-po-files %A %O %B %P"
```
-->
(The proposed merge script seems to be non-functional at this moment)
Whenever you merge, it is best to run `make po` and commit updated po files
just after the merge.
## Debugging PO file contents
Broken build problems for translation are non-trivial to debug by reading the
build log or tracing the intermediary files. I made `bin/xmlpo` script to
check sanity of PO files.
This not only finds invalid XML but also find unmatched XML tags which may be
caused by careless unfuzzy.
```sh
$ bin/xmlpo -i po4a/po/ja.po
E: line=3087 **UNMATCHED XML TAG: fuzzy=False tags_msgid=Counter({'ulink': 33}) tags_msgstr=Counter({'ulink': 32})
E: line=3349 **UNMATCHED XML TAG: fuzzy=False tags_msgid=Counter({'emphasis': 1}) tags_msgstr=Counter()
E: line=3368 **UNMATCHED XML TAG: fuzzy=False tags_msgid=Counter({'emphasis': 4, 'ulink': 1}) tags_msgstr=Counter({'emphasis': 3, 'ulink': 1})
E: line=6869 **UNMATCHED XML TAG: fuzzy=False tags_msgid=Counter({'literal': 3}) tags_msgstr=Counter({'literal': 2})
E: line=7377 **UNMATCHED XML TAG: fuzzy=False tags_msgid=Counter({'literal': 3}) tags_msgstr=Counter({'literal': 2})
...
ERROR COUNT = 43
```
### Set up a local git repo with remotes `salsa` and `weblate`
```sh
$ git clone -o salsa git@salsa.debian.org:debian/debian-reference.git
$ cd debian-reference
$ git remote add weblate https://hosted.weblate.org/git/debian-reference/translations/
$ git remote update weblate
$ git config merge.merge-po-files.driver "./bin/merge-po-files %A %O %B %P"
```
There are three branches now:
- `latest` is local branch which uses salsa's `latest` as upstream
- `salsa/latest` is remote tracking branch for `salsa`'s `latest`
- `weblate/latest` is remote tracking branch for `weblate`'s `latest`
### Set up CLI `cli` for `weblate` service
Install CLI:
```sh
$ sudo apt install wlc
```
Register account in weblate and get your API key from:
* https://hosted.weblate.org/accounts/profile/#api
Put your API key in your personal weblate conf file: `~/.config/weblate`
```
[keys]
https://hosted.weblate.org/api/ = ***your API key strings***
```
### Check and update weblate
Let's check weblate's situation.
```
$ wlc repo
needs_commit: True
needs_merge: False
needs_push: False
url: https://hosted.weblate.org/api/projects/debian-reference/repository/
```
If you see `needs_commit: True`, execute following to update weblate git repo
from translators current works:
```
$ wlc commit
```
Then, you should see:
```
$ wlc repo
needs_commit: False
needs_merge: False
needs_push: False
url: https://hosted.weblate.org/api/projects/debian-reference/repository/
```
You can get the latest committed git repo contents on `weblate` as follows:
```
$ git remote update weblate
```
### Check and update salsa
You can get the latest committed git repo contents on `salsa` as follows:
```
$ git remote update salsa
```
### Merge simple
When `weblate` is a linear decedent of `salsa` and your unpublished local
branch is a linear decedent of the latest `salsa`, it's a good idea to rebase
your unpublished local branch to `weblate`.
Let's first verify our situation as:
```
$ git checkout latest
$ git remote update
$ gitk --all
```
If the relationship of branches is as expected, let's rebase:
```
$ git rebase -i weblate/latest
... resolve conflicts
```
If source tree situation is in good shape after build test, let's push it to
`salsa` and update `weblate` by the updated `salsa` using `wlc`:
```
$ git push
$ wlc pull
```
If `wlc pull` experience time-out (30s?), try `wlc pull` again.
### Merge complication
When `weblate` is not a linear decedent of `salsa` and your unpublished local
branch is a linear decedent of the latest `salsa`, it's a good idea to merge
`weblate` and `salsa` first.
```
$ git checkout latest
$ git tag local-latest
$ git reset --hard salsa/latest
$ git clean -d -f -x
$ git rebase -i weblate/latest
... resolve conflicts
```
If source tree situation is in good shape after build test, let's push it to
`salsa` and update `weblate` by the updated `salsa` using `wlc`:
```
$ git push
$ wlc pull
```
If `wlc pull` experience time-out (30s?), try `wlc pull` again.
Then merge the old local-latest.
```
$ git reset --hard local-latest
$ git clean -d -f -x
$ git rebase -i weblate/latest
... resolve conflicts
```
If source tree situation is in good shape after build test, let's push it to
`salsa` and update `weblate` by the updated `salsa` using `wlc`:
```
$ git push
$ wlc pull
```
## Translation tips
### Sanity check of a PO file:
If you wish to check only "`ja`", you can try following:
```sh
$ make LANGPO=ja check
```
Look for:
* po file syntax
* unquoted `"`
* accidental folding of lines (missing trailing `"`)
* URL style
* `%` in URL string (use unescaped UTF-8 characters, instead)
This check is very useful for identifying the exact root cause of build
failures.
### Remove duplicate entries in a PO file
```
$ mv some.po some.old.po
$ msguniq some.old.po >some.po
````
### Add a new PO
You can create a new template PO file for a new language and edit it as:
```
$ cd po
$ cp templates.pot ??.po
$ poedit ??.po
```
### Merge PO files
This repository now uses very smart 3-way merge for PO files.
Just add a branch for contributed PO branching off from its origin. Then run:
```
$ git checkout latest
$ git merge new-branch
... resolve conflict and commit
$ make po
$ git commit -a -m "make po"
```
When 2 person update a PO file, you need to merge them using tools in the
`gettext` package.
## Test build tips
### Full build
You can test build the whole tree in `testing`/`sid` environment as:
```
$ sudo apt-get update
$ sudo apt-get install devscripts dctrl-tools wget git gitk
$ sudo apt-get build-dep debian-reference
$ cd /path/to/debian-reference
$ debuild
...
```
Please note this takes very long time.
It may be wise to create some virtual build environment while avoiding to
update your main system.
### Partial build after translation
If you just translated a PO file, e.g. `po/it.po`, please
reformat and test it first by generating HTML files only:
```
$ make wrap; make po
$ make test "LANGPO=it"
```
You have English and Italian HTML files under the `build/` directory.
Specifying "`LANGPO`" variable as above saves build time by skipping other
languages.
### Partial build after minor English updates
If you just updated minor English typos in `rawxml/*_*.rawxml`, please test it
first by generating HTML files only:
```
$ make test "LANGPO="
```
You have English HTML files under the `build/` directory.
If this test causes no error, clean the source tree and commit it with:
```
$ make clean
$ git add rawxml/*_*.rawxml
$ git commit rawxml -m "Updated ...."
$ make wrap; make po; make clean
$ git commit -a -m "make po"
```
Please note many PO files are changed in the way `git` doesn't merge cleanly.
### Full build after major English updates
For major English updates in `rawxml/*_*.rawxml` with new URL references or
new packages, please `distclean` and `entity` first to update the package
database.
```
$ make distclean
$ make entity
```
The rest are the same as the minor English updates.
### `@-@` in `*_*.rawxml`
You see quite a bit of `@-@` in `rawxml/*_*.rawxml` and its derivative
`rawxml/debian-reference.rawxml` source. That is placed there when
auto-generated data replaces them as entity in the final form (`*.xml`,
`*.po`, `*.html`, ...).
For example:
* `@-@foo@-@` is converted to the entity definition &foo;
* `@-@popcon1@-@` is converted to popcon value and URL link of package in column 1.
* `@-@psize1@-@` is converted to size value and URL link of package in column 1.
* `@-@popcon2@-@` is converted to popcon value and URL link of package in column 2.
* `@-@psize2@-@` is converted to size value and URL link of package in column 2.
These are transformed into XML when conversion from
`rawxml/debian-reference.rawxml` to `en.xml` and `en.xmlt` happens.
## URL in the English source and PO files
When selecting URL in the English source, please make sure to pick an
internationalized URL which understands the browser language preferences.
## Not use "%" in URL
If URL uses UTF-8 characters, use original ones instead of ones with "%". (If
"%" exists in PO file data, it will break the build.) The proper PO file data
rule is simpler to explain by an example. For example, French translation for
`https://en.wikipedia.org/wiki/Character_encoding` can be found at
`https://fr.wikipedia.org/wiki/Codage_des_caractères` but copying its URL box
in the browser yields percent encoded string
`https://fr.wikipedia.org/wiki/Codage_des_caract%C3%A8res`.
### Best PO file entry format:
Use the UTF-8 character.
```
msgid "https://en.wikipedia.org/wiki/Character_encoding"
msgstr "https://fr.wikipedia.org/wiki/Codage_des_caractères"
```
### Acceptable PO file entry format:
Use entity reference.
```
msgid "https://en.wikipedia.org/wiki/Character_encoding"
msgstr "https://fr.wikipedia.org/wiki/Codage_de_caract%C3%A8res"
```
### Illegal and unacceptable PO file entry format (I.e., Please don't copy-and-paste):
```
msgid "https://en.wikipedia.org/wiki/Character_encoding"
msgstr "https://fr.wikipedia.org/wiki/Codage_des_caract%C3%A8res"
```
Please note most wikipedia pages have link to the translated text at the left
of web page.
https://www.urldecoder.io/ can help to transfer the URL to UTF-8 online.
## Don't translate wrong part
Do not change "`&foo;`" in the `msgstr` of a PO file for translating URL
references. Look for the top part of the PO file defining the "`foo`" entity.
## Don't translate the XML tag name itself
You may see XML tags such as `<literal>` and `<ulink url=\"&gnome;\">` in
`msgid`. Don't change them in translation in `msgstr`.
## Don't translate the string marked up with `<literal>....</literal>`
You may see string such as `<literal>foo</literal>`. If `foo` in your
language means `bar`, please translate this as: `<literal>foo</literal>
(bar)`. This is because `foo` here is a literal string matching the string in
the configuration file.
## Hint for debugging PO file.
If "`make test LANGPO=your_language`" fails while processing PO file but it is
not obvious where is the sorce of problem from he error message, execute
"`make wrap LANGPO=your_language`". Then you may get better error message to
identify the source of problem.
If the above works and "`make test LANGPO=`" to build English HTML works
without problemi, then try "`make test LANGPO=your_language`". If it fails
while making translated XML file, you have a bit more complicated problem.
You may see "Error 255" message like:
```
$ make test LANGPO=de
...
No candidate position for the addendum /home/osamu/salsa/debian-reference/debian-reference/po-tmp/de.add.
Addendum po-tmp/de.add does NOT apply to debian-reference.de.xml (translation discarded).
make: *** [Makefile:462: xml] Error 255
```
(Mostlikely, your `debian-reference.de.xml` is much smaller than `debian-reference.en.xml`.)
If this happens, remove offending `po/*.add` file and build the source again.
Then you see different error message.
```
$ make test LANGPO=de
...
mkdir -p /home/osamu/salsa/debian-reference/debian-reference/build
xsltproc --novalid --nonet --xinclude --stringparam base.dir /home/osamu/salsa/debian-reference/debian-reference/build/ \
--stringparam html.ext .de.html \
xslt/style-html.xsl debian-reference.de.xml
debian-reference.de.xml:1940: parser error : Premature end of data in tag section line 1917
^
unable to parse debian-reference.de.xml
make: *** [Makefile:495: /home/osamu/salsa/debian-reference/debian-reference/build/index.de.html] Error 6
```
This means:
* `debian-reference.de.xml` has 1940 lines and stopped due to error.
* Some error happened somewhere between 1917 and 1940 in
`debian-reference.de.xml`
* Look for matching content of line near 1917 and 1940 in `po/de.po`
* Error is most likely, missing `<placeholder type=\"footnote\" id=\"0\"/>`
etc.
## Flow chart for the building of this documentation
The English source is kept as rawxml/*_*.rawxml files under rawxml/.
The translation source is kept as PO files under po/.
```
>>> rawxml/*_*.rawxml remotely fetched data files
$(RAWXML) $(REMOTE_DATA)
========= ^^^^^^^^^^^^^^^^^^^^^^^^^^^
| :
| :
debian-reference.rawxml :
| :
| :
+---> pkg.lst -------------------> pkgsize.ent, popcon.ent, common.ent
| $(RAWENT) (+)
| ========
| |
+---> $(DRAW)/header.rawxml |
| | |
+<--------+ |
| |
v |
+---------------------------+ |
| | |
v v |
debian-reference.en.xmlt debian-reference.en.xml |
| (**) | | |
v | | |
po/templates.pot (old) | | |
po/ja.po (old) | | |
| | | |
[po4a with po4a.cfg] | | |
| | | |
po/templates.pot (new) | | |
po/ja.po (new) | | |
| | | |
| (debian-reference.ja.xmlt)| | |
| ^^unsed data generated^^ | | |
| | | |
[cp -r po po-tmp] | | |
| | | |
po-tml/templates.pot | | |
po-tmp/ja.po | | |
| | | |
| | | |
v | | |
[po4a with po4a-tmp.cfg] <--------+ | |
| | |
| | |
debian-reference.ja.xml generated | |
| (fuzzy.log) (++) | |
| | |
v v v
+----------------------------------------------+----------+
|(make all), (make test),
|(make html), (make txt), (make pdf), ...
v
all documents (*.*.html *.*.txt *.*.pdf)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
Actual build process builds `po4a.cfg` and `po4a-tmp.cfg` from `po4a.cfg.in`
and `po4a-tmp.cfg.in` to reduce workload by limitting run for active
translations. Language listed in `Makefile` can be added whenever any
translator feels needed. But adding languages to `debian/rules` and
`debian/control` needs to be discussed with the Debian packaging owner.
Chinese translation uses OpenCC where `cp -r ...` is used.
```
# RAWXML updated with reference to new packages
$ make entity # update all RAWENT with REMOTE_DATA (NET)
# RAWXML updated without reference to new packages
$ # no action needed
# before PO update for LANGPO=XX
$ make po LANGPO=XX # update all PO in sync with RAWXML and RAWENT
# after PO update for LANGPO=XX
$ make po LANGPO=XX # normalize PO
# clean source for commit (except ===== data)
$ make clean
# clean source for commit (except ===== and ^^^^^^^ data) (Keep remote data)
$ make localclean
# clean source including entity (except ===== and ^^^^^^^ data) (Keep remote data)
$ make entityclean
# build package
$ make all # build all documents
# test build English html (keep REMOTE_DATA)
$ make en
# test build LANGPO=XX html
$ make test LANGPO=XX
# no wrap po file for LANGPO=XX
$ make nowrap LANGPO=XX
# wrap po file for LANGPO=XX
$ make wrap LANGPO=XX
# replace string with 'sed -f foo' over po file for LANGPO=XX
$ make replace LANGPO=XX SEDF=foo
```
* (`+`) When the English source changes URL link location or packages
mentioned in the package list table, you need to update `*.ent` files in the
source by running "`make entity`" with remote file access.
* (`**`) `debian-reference.en.xmlt` effectively replaces untranslatable
contents in `debian-reference.en.xml` with "DUMMY" to reduce clutter in PO
files.
* (`++`) `fuzzy.log` file contains number of fuzzy translation. If not zero,
please manually update PO file contents.
### Build failure due to po/*.add
If `po/foo.add` can't locate a matching string, build fails. Since no
`debian-reference.foo.xml` for the problematic language is build, it's a bit
difficult to debug situation. If you face such problem, please rename such
`po/foo.add` to something like `po/foo.add-keep`, and execute
"`make LANGPO=foo html`" to make `debian-reference.foo.xml`. Then you can
look for the location of interest and make it the matching string. Then
rename `po/foo.add-keep` back to `po/foo.add` and update the matching string.
It's tricky. (This should be less problematic after recent po4a updates)
## Publication to the www.debian.org servers
The contents of `debian-reference` are published to the www.debian.org servers
and they are linked from:
* https://www.debian.org/doc/user-manuals#quick-reference
These are generated from the latest released packages by the `cron` script run
on the master www.debian.org server and they are mirrored to all the
www.debian.org servers.
* https://salsa.debian.org/webmaster-team/cron
* `parts/1ftpfiles`
* `parts/7doc`
## Is there easier way to edit XML source
Unfortunately, no.
## Reminder
Updating the `*.ent` files in the source tree require you to install `wget`
and `dctrl-tools` and run "make entity". This must be done before the source
package upload in manual process. This part of code can't be run as a part of
automatic package build process since package build should not access remote
data by Debian Policy.
Osamu
Generated by dwww version 1.16 on Sat Oct 3 02:28:10 CEST 2026.